--- name: "skill_code_ai_md_power" description: "Processes QEPM power CSV and modem mdlog into aligned 1s outputs. Invoke when handling AI modem power breakdown cases, mixed power+modem folders, or rebuilding mdlog-derived CSVs." version: "1.1.14" --- # Skill Code AI MD Power This skill standardizes the AI modem power breakdown workflow for N1 and similar projects. ## Language Policy Default to Chinese for all user-facing reports, analysis notes, comparison documents, conclusions, and explanatory text under `references/`. Use English only when the content itself is source material or code-level material: - code, generated code comments, CLI options, file names, paths, API names, config keys, and schema fields - original log lines, protocol field names, model feature names, metric names, and copied upstream document titles - table cells that must preserve exact source values for reproducibility For training and evaluation reports, metric explanations must be beginner-friendly and must include units in table headers when applicable, such as `RMSE (mW)` and `MAE (mW)`. Do not write default report sections in English unless the user explicitly asks for an English report. It covers six linked steps: 1. Run preflight time-range overlap checks before expensive processing. 2. Convert raw QEPM power CSV into normalized cellular power stats. 3. Enforce the `power_stats.csv` completeness gate before any mdlog parsing or modem merge. 4. Parse raw modem mdlog into `auto_analysis` CSV outputs when needed, then merge modem CSVs by template and rules. 5. Build the `COM_DATA_PHY` reference end-window, align both power and modem data by `[t-0.5s, t)`, then write the final 1-second averaged CSV. 6. Optionally align AP-side `PowerConsumptionAnalyzer` 1-second parameters with the same power breakdown, then generate reusable HTML visualizations when requested. ## Mandatory Execution Order Always follow this order for every case: 1. Classify the input layout and choose the correct entry. 2. Run preflight time-range overlap checks. 3. Generate `runs/_v/tmp/power_stats.csv`. 4. Enforce the power completeness gate on `SOC_MODEM`, `RF_Tranceiver`, `RF_ET_DCDC`, and `RF_TRX`. 5. For raw modem logs, validate qdss presence, qdss size, and qdss-vs-power time overlap before mdlog parsing. 6. Only if the power gate and raw qdss validity gate both pass, continue to mdlog parsing or modem CSV merge. 7. Align both sides by the `COM_DATA_PHY` reference end-window and export the intersected 1-second averaged timeline. 8. Keep all run-generated artifacts inside the same `runs/_v/`. Never start raw mdlog parsing first just because the directory layout is ambiguous or mixed. The power side must be proven usable before spending time on mdlog parsing. Also, never treat `qmdl` or `qmdl2` as the modem main-log source of truth. They are qcril-side logs. Raw modem validity must be judged from `qdss`. ## When To Invoke Invoke this skill when the user wants to: - process a modem power breakdown case from raw logs - combine power CSV with modem mdlog or modem CSV - rebuild `*_power_modem_1s_combined_utc8_with_totals.csv` - plot aligned power/AP/modem CSV columns into an interactive HTML chart - batch-run multiple modem power breakdown cases - reduce trial-and-error caused by mixed input layouts ## Source Of Truth - Power template: `scripts/2_data_process_tool/config/n1/N1_PQ85A02_功耗拆解数据处理模板V1.2_增加通讯模块统计.xlsx` - Network template and rules: `scripts/2_data_process_tool/config/network_data_source_rules_compact.csv` - Main combiner: `scripts/2_data_process_tool/power_modem_combiner/combine_power_modem.py` - Structured scenario entry: `scripts/2_data_process_tool/main.py` - Ocean FTP staging helper (canonical owner): `../skill_log_ocean_ftp/scripts/prepare_ocean_ftp_power_cases.py` - AP vs modem throughput plot entry: `scripts/2_data_process_tool/tput_compare_plot/plot_ap_modem_tput_compare.py` - Offline model training entry: `scripts/4_offline_model_train/src/train/train.py` - Offline model set: `RF`, `GBDT`, `XGB`, `MLP`, and `BREN` - Offline grouped dataset builder: `scripts/4_offline_model_train/src/data/build_valid_grouped_dataset.py` - Offline default training dataset: `scripts/4_offline_model_train/res/datasets/datasets_valid_grouped_by_case.csv` ## Offline Model Evaluation Split For cellular power model evaluation, the minimal invariant is: ```text train.source_case_id ∩ test.source_case_id == ∅ ``` Do not treat row-level random split metrics as real generalization metrics when source case metadata is available. Adjacent or same-case samples share network state, RF condition, traffic pattern, and parameter space. If rows from one case enter both train and test, the test set leaks part of the answer even when every individual row appears in only one side. Default workflow for `resources/3_valid_dataset/split_by_rat_band`: 1. Build the grouped dataset first: ```bash python src/data/build_valid_grouped_dataset.py ``` 2. Confirm the summary prints `overlap=0`. 3. Keep `res/datasets/datasets_list.txt` pointed to `datasets_valid_grouped_by_case.csv`. 4. Run training: ```bash python src/train/train.py ``` 5. Report case-level grouped holdout metrics as the primary result. The training loader must also enforce the invariant at load time. If `source_case_id` exists and any case appears in both train and test, stop and fix the dataset instead of training. Row-level split results may be kept only as historical or debugging references, and must be explicitly labeled as leakage-prone and optimistic. ## Terminology Avoid using the standalone word "organize" for this workflow because it is ambiguous. Use these stage-specific terms instead: - Input staging: reshape a raw or mixed case into a stable runnable layout such as `input/` - Run-scoped output: before processing starts, create `runs/_v/` and write this run's `tmp/`, `output/`, and generated `auto_analysis/` there directly When the user says the output is already complete and wants to prepare for later runs, interpret that as output archiving, not input staging. ## Input Routing Always classify the input first. ### Type A: Structured scenario root Directory shape: ```text / input/ qepm_raw_xxx.csv auto_analysis/ *.csv runs/ _v/ output/ tmp/ ``` Use: ```bash python scripts/2_data_process_tool/main.py -d /input ``` This is the preferred stable entry. `main.py` also accepts `` directly and will auto-route to `/input` when it already exists. If the provided path is any descendant under `/input/`, such as `/input/log_xxx`, the run root remains `` and all generated artifacts must be written under `/runs/_v/`. ### Type B: Raw modem zip Example: ```text case_xxx/ qepm_raw_xxx.csv raw_log.zip ``` Use `combine_power_modem.py` with `--modem-zip`. ### Type C: Raw `diag_logs/modem` Example: ```text .../diag_logs/modem/ ``` Use `combine_power_modem.py` with `--modem-input-dir <.../diag_logs/modem>`. Execution order remains strict: - preflight time-range check - generate and validate `power_stats.csv` - validate raw qdss quality and qdss-vs-power overlap - only then call `skill_log_filter_mdlog` - continue modem merge and final align ### Type D: Mixed root folder Example: ```text log1/ power.csv modemdebug/ log_xxx/ diag_logs/ modem/ ``` This is the main error-prone shape. Rules: - `main.py -d ` is now allowed and preferred when you want the script to auto-route and create `runs/_v/tmp|output`. - `combine_power_modem.py` still accepts explicit `--power-input` and `--modem-input-dir` for advanced control. - In mixed-root cases, always prove the power side first by generating and validating `power_stats.csv` before any mdlog parsing starts. The combiner now contains a guard: - if a directory contains both plain CSV files and nested `diag_logs/modem`, it prefers the mdlog parsing path instead of treating root CSV files as modem CSV inputs ## Recommended Invocation Order 1. Prefer `scripts/2_data_process_tool/main.py` for both structured cases and mixed raw case roots when you want the script to auto-route inputs and create `runs/_v/tmp|output`. 2. Use `power_modem_combiner/combine_power_modem.py` when you need explicit control over `--power-input`, `--modem-input-dir`, or `--modem-zip`. 3. Always let the pipeline finish preflight and the `power_stats.csv` completeness gate before any mdlog parsing or modem merge. 4. For raw mdlog, let the combiner validate qdss first; only then drive `skill_log_filter_mdlog`. 5. Only pass `--modem-inputs` when you already know the exact modem CSV list. 6. When the user requests AP-vs-modem throughput HTML analysis, use `scripts/2_data_process_tool/tput_compare_plot/plot_ap_modem_tput_compare.py` after `ap_power_consumpution.csv` and `modem_merged_1s.csv` are ready. ## Mdlog Completion Decision Do not use directory names such as `temp_*`, `auto_analysis`, or `backup` as the only completion signal. The real gate is whether the current modem parse snapshot is already sufficient for the target power overlap window. Important distinction: - preflight may use a modem filename grace window to decide whether power and raw mdlog are allowed to enter parsing - consumable-snapshot judgment must use the real raw-mdlog filename upper bound, not the grace-extended upper bound Otherwise the parser may already have produced a stable `auto_analysis` snapshot, but the outer workflow will keep waiting for an impossible tail interval that no raw mdlog file can actually cover. Always evaluate raw-mdlog progress in this order: 1. Determine the power-side overlap window first, especially `power_overlap_start` and `power_overlap_end`. 2. Cap the mdlog consumption end time by the real raw mdlog filename end when it is earlier than the power overlap end. 3. Identify the minimum required modem CSV set for the requested final columns. 4. Check time coverage of the currently parsed modem CSVs against that capped consumption window. 5. Check progress signals such as parser process liveness, file count growth, row count growth, and latest parsed timestamp growth. 6. Only then decide whether to keep waiting, allow an early stop, or mark the run incomplete. Treat `temp_*` as an intermediate snapshot by default, not as an automatic failure and not as an automatic success. ### Continue Waiting Keep waiting when any of these is true: - the parser process is still running - a required mask or required modem CSV is still missing - required masks exist but their covered end time is still earlier than the capped mdlog consumption end - files, rows, or latest parsed timestamps are still growing inside the relevant output directory ### Allow Early Stop And Consume Current Results You may stop waiting and consume the current modem snapshot only when all of these are true: - the required mask set for the requested final output is already complete - the required modem CSV time coverage already spans the capped mdlog consumption window - any still-missing non-required masks do not feed the requested final columns - further parser progress would only add data strictly after the capped mdlog consumption end - a short stability window confirms no new effective coverage is being added Recommended stability window: - observe for about `30-60s` - if file count, row count, and latest effective timestamp all stay unchanged, treat the snapshot as stable ### Mark As Incomplete Or Escalate Mark the run as incomplete, or switch to manual recovery, when any of these is true: - the parser process has exited or stalled, and required coverage still does not reach `power_overlap_end` - the parser process has exited or stalled, and required coverage still does not reach the capped mdlog consumption end - only partial masks are available and the missing masks feed requested final columns - the current modem snapshot cannot explain the power overlap window even after the stability check In that state, do not present the combined result as final. ## Common Mistakes - Passing a mixed case root as `--modem-input-dir` without checking whether root-level CSVs are power files. - Assuming any `.csv` under a case directory is modem data. - Starting mdlog parsing before checking whether `power_stats.csv` is usable. - Assuming `qmdl` or `qmdl2` alone prove raw modem logs are valid. The main modem log is `qdss`. - Ignoring obviously invalid qdss captures such as missing qdss files, qdss files with only a few KB, or qdss filename time ranges that do not overlap the power capture. - Assuming `unknown_masks` should stop the run immediately. In this skill they remain non-blocking because some columns may be optional, derived elsewhere, or pending parser completion. - Treating `temp_*` as automatically final or automatically invalid without checking coverage, required masks, and progress signals. ## Deliverables Expected outputs: - During execution, this run writes directly into `runs/_v/tmp`, `runs/_v/output`, and run-scoped `auto_analysis/` - `runs/_v/tmp/power_stats.csv` - `runs/_v/tmp/modem_merged_1s.csv` - `runs/_v/output/*_power_modem_1s_combined_utc8_with_totals.csv` - `runs/_v/output/*_power_aplog_1s_combined_utc8_with_totals.csv` when `*predicted_power_lines*.txt` is provided or discovered - `runs/_v/output/*.html` for requested reusable visualization charts - `runs/_v/.../auto_analysis/*.csv` when the source is raw mdlog - `runs/_v/run_manifest.json` records the run time, skill version, CLI arguments, and archived paths ## Run Archiving Run-scoped output follows these invariants: - The run directory is created before heavy processing starts, not after the run finishes. - `input/` remains the source of truth and must not be rewritten as part of result retention - Generated artifacts should land in the active run directory directly, including `output/`, `tmp/`, and `auto_analysis/` produced from raw mdlog during this run - `*/input/*` paths never own output state; they are input descendants only. Use the parent of `input/` as the case root, or pass `--run-dir /runs/` for an explicit run. - If the user directly provides parsed modem CSV under `input/auto_analysis`, that directory is treated as input and is not moved - The archive directory name includes local run time and skill version so repeated runs with unchanged input remain traceable - `run_manifest.json` is the machine-readable record for answering “when was this run generated” and “which skill version produced it” ## Reusable Visualization Use this section when the user asks to draw curves from aligned CSV outputs such as: - `/*_power_modem_1s_combined_utc8_with_totals.csv` - `/*_power_aplog_1s_combined_utc8_with_totals.csv` Always follow these invariants: - Read the CSV header first and use real column names from the file instead of trusting user spelling. - Build the x-axis from `Date` + `Time` when both columns exist; otherwise use the available timestamp column. - Keep output as standalone interactive HTML unless the user requests another format. - Place the HTML next to the input CSV and use a descriptive name based on the plotted columns. - Include row count, time range, and valid-point counts for every plotted curve in the page header. Common dual-axis plot for AP predicted power validation: - Input: `*_power_aplog_1s_combined_utc8_with_totals.csv`. - X-axis: time from `Date` + `Time`. - Left Y-axis: power in `mW`. - Right Y-axis: transmit power in `dBm`. - Left-axis curves: `NETWORK_TOTAL` and `AP_Predicted_Power`. - Right-axis curve: `PUSCH_TxPower`. - Note: the column is `NETWORK_TOTAL`; do not use the common typo `NETWORK_TOAL`. Plotly layout guidance: - Prefer external HTML section titles over internal Plotly titles when the page already has a header or chart card title. - Keep context metadata such as `RAT`, `BAND`, `Freq_DL`, and `SCS` outside the Plotly plotting area as HTML badges or tables; keep the same fields in hover data. - Do not place Plotly `title`, horizontal `legend`, and top `annotations` in the same top margin area. ## Batch ICD Compare Refresh When the user wants to regenerate `offline vs online` ICD HTML compare artifacts for multiple existing runs, treat that as `skill_code_add_qcom_icd` ownership, not `skill_code_ai_md_power` ownership. Reason: - the source-of-truth compare logic lives in `skill_code_add_qcom_icd/scripts/icd_offline_online_compare.py` - the batch task is about regenerating ICD compare artifacts, not rebuilding the power/modem pipeline itself Use the official batch entry from `skill_code_add_qcom_icd`: ```bash python ../skill_code_add_qcom_icd/scripts/rerun_icd_compare_batch.py \ --root ``` Keep `skill_code_ai_md_power` focused on producing run-scoped inputs such as: - `tmp/mdlog_auto_analysis_modem/` - `tmp/online_required_input/` - `run_manifest.json` - Put the legend above the plotting area only when enough top margin is reserved; otherwise move the title and context labels outside Plotly. - Limit annotations or reserve dedicated space for them; avoid using top annotations for dense or repeated context labels. - Use SVG `scatter` by default for small and medium CSV plots; use `scattergl` only when WebGL support is known and the point count requires it. - Keep `hovermode: 'x unified'` for time-aligned comparison. - Enable an x-axis range slider for quick zooming across long captures. Specialized `NR_MAC_UL_SCHEDULE.csv` visualization: - Input: `NR_MAC_UL_SCHEDULE.csv`. - Scope: filter `Channel == PUSCH`, then plot `PUSCH_MCS_Table`, `PUSCH_MCS`, `PUSCH_Modulation_Order`, and `PUSCH_Code_Rate`. - X-axis: timestamp from `Date` + `Time`. - Output: a standalone HTML page with 3 vertically stacked plots: 1. `PUSCH_MCS` index curve 2. `PUSCH_MCS_Table` and `PUSCH_Modulation_Order` 3. `PUSCH_Code_Rate` curve, normalized as `Q11 / 1024` while keeping raw `Q11` in hover text - Page header must include source path, SubID, time range, row count, and valid-point counts. Example: ```bash python3 scripts/2_data_process_tool/mdlog_filter/skill_log_filter_mdlog/scripts/mdlog_filter_tool/LogPlot.py \ /path/to/NR_MAC_UL_SCHEDULE.csv \ --subids 1 \ --xtick-sec 1 \ --outdir /path/to/output ``` - Keep curve units explicit in trace names, such as `NETWORK_TOTAL (mW)` and `PUSCH_TxPower (dBm)`. Layout self-check before delivery: - Verify the HTML has no overlapping chart title, legend, annotations, or context labels in the rendered top area. - Verify any internal Plotly title is absent when an external chart title exists. - Verify top annotations are absent or intentionally limited with enough reserved margin. - Verify `scattergl` is not used unless WebGL compatibility is acceptable for the target viewer. - If the plot is split into stacked charts, keep each chart title and context metadata outside the Plotly div. ## Verification Check these invariants: - preflight reports overlapping time ranges before expensive mdlog parsing; exit code `6` means no overlap and downstream evaluation should stop - after `runs/_v/tmp/power_stats.csv` is generated, critical power modules must contain valid samples before any modem parsing starts - if `SOC_MODEM`, `RF_Tranceiver`, `RF_ET_DCDC`, or `RF_TRX` is completely missing, the pipeline must stop immediately with exit code `7` - power side has non-zero 1-second bins - modem side produces a consumable snapshot whose required masks and time coverage already satisfy the power overlap window - final align step reports non-zero intersection rows when time ranges overlap - output header follows the network rules template plus power summary columns - AP-side output is generated only when `PowerConsumptionAnalyzer: Updated signal info` lines are parsed and overlap the power timeline - visualization outputs must preserve the requested axis ownership and report valid-point counts before interpreting curve differences - visualization outputs must pass the layout self-check and avoid title/legend/annotation overlap before delivery ## Preflight Time Range Check - QEPM raw CSV range is derived from first-line start time plus the `Time Offset` column. - Raw mdlog range is derived from mdlog filenames containing `YYYYMMDD_HHMMSS`; this filename source is treated as local comparable time and is not shifted by `--modem-utc-offset-hours`. - Parsed modem CSV range is derived from `Timestamp` or `Date` + `Time`; this source still uses `--modem-utc-offset-hours` before comparison. - If both sides are known and do not overlap, `combine_power_modem.py` exits with code `6` before power conversion, mdlog parsing, modem merge, or final evaluation. - Use `--skip-preflight-time-range` only when the case has a known non-standard time source and must force the legacy full pipeline. ## Power Completeness Gate - The pipeline now adds a hard gate immediately after `power_stats.csv` is produced. - This gate protects against wasting time on mdlog parsing and modem merge when the power breakdown itself is already unusable. - The gate checks critical module columns: `SOC_MODEM`, `RF_Tranceiver`, `RF_ET_DCDC`, `RF_TRX`. - If any of these columns has zero valid numeric samples across the whole `power_stats.csv`, the tool stops and asks the user to inspect the power side first. - The failure log must include the `power_stats.csv` path, the missing module names, and per-column valid-point counts.