# LLMS-FULL: Trading Ecosystem Rispecs
# Generated: Sun 01 Feb 2026 07:42:08 AM EST
# ==========================================
# This file contains all rispecs linked from @rispecs/
# CNAME-resolved paths enable agent URL resolution


===============================================
FILE: ecosystem.spec.md
PATH: rispecs/ecosystem.spec.md
===============================================

# JGT Trading Ecosystem - Master Specification

> Unified RISE specification for the Williams-Native Trading Intelligence Platform

**Specification Version**: 1.0  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: A complete trading intelligence platform that transforms raw market data into validated trading signals through Bill Williams' 5 Dimensions methodology, served through CLI tools, REST APIs, and AI-powered terminal agents.

**Achievement Indicator**: Traders can discover FDB signals, validate them against higher timeframe trends, execute trades with proper risk management, and track performance through a unified ecosystem of interconnected packages.

**Value Proposition**: Replace fragmented trading tools with a coherent system where Williams methodology flows natively through every layer, from price data to trading decisions.

---

## Ecosystem Structural Tension

**Current Reality**: Trading decisions require consulting multiple disconnected tools, manual data processing, and subjective interpretation of market conditions.

**Desired State**: Unified ecosystem where:
- Data flows automatically from broker to analysis
- Signals are validated against multiple timeframes
- Three-universe perspectives provide coherent analysis
- AI agents guide traders through Medicine Wheel phases
- Everything speaks Williams methodology natively

**Natural Progression**: Each package builds upon the previous, creating structural tension that resolves toward validated trading decisions.

---

## Package Specifications Matrix

| Layer | Package | Core Purpose | Key Specs |
|-------|---------|--------------|-----------|
| **Core** | jgtcore | Configuration | `get_config()`, `get_settings()`, `setup_environment()` |
| **Utility** | jgtutils | CLI + Helpers | `jgtset`, `jgtutr`, `tfw/wtf` |
| **Indicator** | jgtapy | Williams Indicators | `Indicators.alligator()`, `.awesome_oscillator()`, `.fractals()` |
| **Broker** | jgtfxcon | Price Data + Orders | `getPH()`, `create_entry_order()` |
| **Data** | jgtpy | Signal Pipeline | `cdscli`, PDS→IDS→CDS→ADS |
| **Analysis** | jgtml | Signal Validation | `fdbscan`, `alligator_cli`, `ttfcli/mlfcli/mxcli` |
| **Insight** | jgt-insight | Regime/Scoring | Regime detection, signal scoring 0-100, correlations |
| **Charting** | Caishen | Interactive Charts | SE, BITT, trading campaigns, signal overlays |
| **API** | jgt-data-server | REST + MCP | `/api/v1/williams/*`, `/api/v1/perspective/*` |
| **Agent** | jgt-code | Terminal UI | Three-Universe Processing, Medicine Wheel |

> **Note**: jgt-insight specs at `ee4c3e7a-5e60-4caf-ba7d-45af1fb9f8c4/rispecs/`  
> **Note**: Caishen rispecs at `caishen/rispecs/`

---

## Williams 5 Dimensions Implementation

### Dimension 1: Fractal (Space)

**Definition**: 5-bar patterns marking potential reversal points

**Implementation Path**:
```
jgtapy.Indicators.fractals()
    ↓ adds fh, fl columns
jgtpy.JGTIDS
    ↓ enriches with ao peak fractals
jgtpy.JGTCDS
    ↓ adds FDB signals (fdbb, fdbs)
jgtml.fdb_scanner
    ↓ validates against HTF
jgt-data-server
    ↓ serves /williams/dimensions
jgt-code
    ↓ displays in Signal Detection universe
```

### Dimension 2: Momentum (AO)

**Definition**: SMA(Median, 5) - SMA(Median, 34)

**Implementation Path**:
```
jgtapy.Indicators.awesome_oscillator()
    ↓ adds ao column
jgtpy.JGTIDS
    ↓ adds aocolor (green/red)
jgtpy.JGTCDS
    ↓ adds saucer signals (sb, ss)
jgtml
    ↓ HTF AO alignment analysis
jgt-data-server
    ↓ serves /williams/ao
```

### Dimension 3: Acceleration (AC)

**Definition**: AO - SMA(AO, 5)

**Implementation Path**:
```
jgtapy.Indicators.accelerator_oscillator()
    ↓ adds ac column
jgtpy.JGTIDS
    ↓ adds accolor
jgtpy.JGTCDS
    ↓ adds acceleration signals (acb, acs)
jgt-data-server
    ↓ part of dimensions response
```

### Dimension 4: Zone Trading

**Definition**: AO + AC color confluence

**Implementation Path**:
```
jgtpy.JGTCDS
    ↓ calculates zone (green/red/gray)
    ↓ adds bz, sz signals
jgtml
    ↓ zone confluence scoring
jgt-data-server
    ↓ serves /williams/zones
```

### Dimension 5: Balance Line (Alligator)

**Definition**: Three SMAs with different periods and shifts

**Implementation Path**:
```
jgtapy.Indicators.alligator()
    ↓ adds jaw, teeth, lips
jgtpy.JGTIDS
    ↓ mouth state analysis
jgtml.TideAlligatorAnalysis
    ↓ Regular (5-8-13) + Big (34-55-89) + Tide (144-233-377)
jgtml.alligator_cli
    ↓ convergence analysis
jgt-data-server
    ↓ serves /williams/alligator
jgt-code
    ↓ displays in Wave Analysis universe
```

---

## Three-Universe Architecture

### Signal Detection Universe 📊

**Focus**: FDB scanning, zone analysis, fractal patterns

**Packages Involved**:
- jgtpy/JGTCDS - Signal generation
- jgtml/fdb_scanner - Signal detection
- jgt-data-server - Signal API

**Key Questions**:
- Is there a valid FDB signal?
- What is the zone confluence?
- Are fractals confirming direction?

### Wave Analysis Universe 🌊

**Focus**: HTF alignment, Elliott Wave position, trend strength

**Packages Involved**:
- jgtml/TideAlligatorAnalysis - Multi-period Alligator
- jgtml/ttfcli - Cross-timeframe features
- jgt-data-server - Perspective API

**Key Questions**:
- Is HTF Alligator aligned with signal?
- Where are we in the larger wave structure?
- Is the trend accelerating or decelerating?

### Coordination Universe 🎯

**Focus**: Risk assessment, position sizing, execution

**Packages Involved**:
- jgtml/SignalOrderingHelper - Risk calculation
- jgtfxcon/jgtfxtransact - Order execution
- jgt-code - Execution guidance

**Key Questions**:
- What is the risk:reward ratio?
- Where is the stop loss (fractal level)?
- How does this fit the trading plan?

---

## Medicine Wheel Trading Integration

| Direction | Trading Phase | Primary Tools |
|-----------|--------------|---------------|
| **East 🌅** | Signal Detection | `fdbscan`, CDS signals |
| **South 🌱** | Validation & Growth | `alligator_cli --type all`, TTF |
| **West 🌊** | Risk Reflection | `SignalOrderingHelper`, position sizing |
| **North ❄️** | Execution Wisdom | `jgtfxtransact`, order management |

---

## Data Pipeline Specification

### Stage 1: Price Acquisition (PDS)

```
FXCM API → jgtfxcon.getPH() → $JGTPY_DATA/pds/{instrument}_{timeframe}.csv
```

**Columns**: Date, Open, High, Low, Close, Volume

### Stage 2: Indicator Enrichment (IDS)

```
PDS → jgtapy.Indicators → jgtpy.JGTIDS → $JGTPY_DATA/ids/
```

**Added Columns**: jaw, teeth, lips, ao, ac, fh, fl, mfi

### Stage 3: Signal Generation (CDS)

```
IDS → jgtpy.JGTCDS → $JGTPY_DATA/cds/{instrument}_{timeframe}.csv
```

**Added Columns**: fdb, fdbb, fdbs, zlcB, zlcS, bz, sz, sb, ss

### Stage 4: Feature Engineering (TTF/MLF/MX)

```
CDS → jgtml.ttfcli → $JGTPY_DATA/ttf/
CDS+TTF → jgtml.mlfcli → $JGTPY_DATA/mlf/
CDS+TTF+MLF → jgtml.mxcli → $JGTPY_DATA/mx/
```

**TTF Features**: HTF Alligator state, trend direction, multi-TF confluence
**MLF Features**: Lagged values, rolling statistics, change rates
**MX Features**: Complete feature set + target variable for ML

---

## CLI Command Reference

| Command | Package | Purpose |
|---------|---------|---------|
| `jgtcli` | jgtpy | Price/indicator data operations |
| `cdscli` | jgtpy | Signal generation |
| `jgtads` | jgtpy | Advanced charting |
| `jgtservice` | jgtpy | Daemon mode |
| `fdbscan` | jgtml | FDB signal scanning |
| `jgtapp` | jgtml | Main trading application |
| `ttfcli` | jgtml | Cross-timeframe features |
| `mlfcli` | jgtml | Meta lag features |
| `mxcli` | jgtml | Matrix generation |
| `jgtset` | jgtutils | Settings management |
| `jgtutr` | jgtutils | TLID range calculation |
| `tfw/wtf` | jgtutils | Timeframe scheduling |
| `jgtfxcli` | jgtfxcon | Price data fetching |
| `jgt-code` | jgt-code | Terminal agent (Node.js) |

---

## API Endpoint Summary

### jgt-data-server REST API

| Endpoint | Description |
|----------|-------------|
| `GET /api/v1/health` | Health check |
| `GET /api/v1/pds/{instrument}/{timeframe}` | Price data |
| `GET /api/v1/cds/{instrument}/{timeframe}` | Signal data |
| `GET /api/v1/williams/dimensions/{instrument}/{tf}` | All 5 dimensions |
| `GET /api/v1/williams/alligator/{instrument}/{tf}` | Balance Line |
| `GET /api/v1/williams/ao/{instrument}/{tf}` | Awesome Oscillator |
| `GET /api/v1/williams/zones/{instrument}/{tf}` | Zone analysis |
| `GET /api/v1/perspective/{instrument}` | Universe perspective |
| `POST /api/v1/refresh` | Trigger refresh |

---

## Detailed Module Specifications

### jgtml Package (Analysis Layer)

| Spec File | Module | CLI | Purpose |
|-----------|--------|-----|---------|
| [jgtapp.spec.md](../jgtml/rispecs/jgtapp.spec.md) | jgtapp.py | `jgtapp` | **Unified trading CLI** |
| [fdb-scanner.spec.md](../jgtml/rispecs/fdb-scanner.spec.md) | fdb_scanner_2508.py | `fdbscan` | FDB signal scanning and validation |
| [signal-ordering.spec.md](../jgtml/rispecs/signal-ordering.spec.md) | SignalOrderingHelper.py | - | Entry order generation and risk |
| [alligator-analysis.spec.md](../jgtml/rispecs/alligator-analysis.spec.md) | TideAlligatorAnalysis.py | `alligator_cli` | Three-scale Alligator analysis |
| [ttf.spec.md](../jgtml/rispecs/ttf.spec.md) | ttfcli.py, ptottf.py | `ttfcli` | Cross-timeframe features |
| [mlf.spec.md](../jgtml/rispecs/mlf.spec.md) | mlfcli.py, realityhelper.py | `mlfcli` | Meta lag features for ML |
| [mx.spec.md](../jgtml/rispecs/mx.spec.md) | mxcli.py, jtc.py | `mxcli` | ML matrix with target calculation |

### jgtpy Package (Data Layer)

| Spec File | Module | Purpose |
|-----------|--------|---------|
| [cds.spec.md](../jgtpy/rispecs/cds.spec.md) | JGTCDS.py, cdscli.py | FDB signal generation |
| [ids.spec.md](../jgtpy/rispecs/ids.spec.md) | JGTIDS.py | Williams indicator calculations |

### jgtfxcon Package (Broker Layer)

| Spec File | Module | CLI | Purpose |
|-----------|--------|-----|---------|
| [pds.spec.md](../jgtfxcon/rispecs/pds.spec.md) | JGTPDS.py | - | Price data acquisition (core) |
| [jgtfxcli.spec.md](../jgtfxcon/rispecs/jgtfxcli.spec.md) | jgtfxcli.py, JGTPDSSvc.py | `jgtfxcli` | Price data CLI + Service |
| [fxtransact.spec.md](../jgtfxcon/rispecs/fxtransact.spec.md) | jgtfxtransact.py | `fxtr` | Orders/trades viewer |
| [fxaddorder.spec.md](../jgtfxcon/rispecs/fxaddorder.spec.md) | jgtfxentryorder.py | `fxaddorder` | Entry order creation |
| [fxrmorder.spec.md](../jgtfxcon/rispecs/fxrmorder.spec.md) | jgtfxremoveorder.py | `fxrmorder` | Order deletion |
| [fxreport.spec.md](../jgtfxcon/rispecs/fxreport.spec.md) | jgtfxreport.py | `fxreport` | Trade reports |

### jgtapy Package (Indicator Library)

| Spec File | Module | Purpose |
|-----------|--------|---------|
| [indicators.spec.md](../jgtapy/rispecs/indicators.spec.md) | indicators.py | Core Williams indicators |

### jgtutils Package (Utilities)

| Spec File | Module | CLI | Purpose |
|-----------|--------|-----|---------|
| [constants.spec.md](../jgtutils/rispecs/constants.spec.md) | jgtconstants.py | - | Column names and constants |
| [jgtset.spec.md](../jgtutils/rispecs/jgtset.spec.md) | jgtset.py | `jgtset` | Settings export/sync |
| [jgtutr.spec.md](../jgtutils/rispecs/jgtutr.spec.md) | cli_tlid_range.py | `jgtutr` | TLID range calculator |
| [tfw-wtf.spec.md](../jgtutils/rispecs/tfw-wtf.spec.md) | timeframe_scheduler.py | `tfw`, `wtf` | Timeframe scheduling |
| [add-pattern.spec.md](../jgtutils/rispecs/add-pattern.spec.md) | add_pattern_to_home_settings.py | `jgt-add-pattern` | Pattern management |

### jgtcore Package (Foundation)

| Spec File | Module | Purpose |
|-----------|--------|---------|
| [core.spec.md](../jgtcore/rispecs/core.spec.md) | core.py, tracer.py | Configuration and tracing |

---

## For LLM Implementers

This ecosystem specification provides everything needed to re-implement the entire JGT trading platform. Follow this reading order:

1. **Core Foundation**
   - [jgtcore/rispecs/core.spec.md](../jgtcore/rispecs/core.spec.md) - Configuration, tracing
   - [jgtutils/rispecs/constants.spec.md](../jgtutils/rispecs/constants.spec.md) - Column constants

2. **Indicator Layer**
   - [jgtapy/rispecs/indicators.spec.md](../jgtapy/rispecs/indicators.spec.md) - Williams indicators

3. **Data Layer**
   - [jgtfxcon/rispecs/pds.spec.md](../jgtfxcon/rispecs/pds.spec.md) - Price data core
   - [jgtfxcon/rispecs/jgtfxcli.spec.md](../jgtfxcon/rispecs/jgtfxcli.spec.md) - Price data CLI
   - [jgtpy/rispecs/ids.spec.md](../jgtpy/rispecs/ids.spec.md) - Indicator enrichment
   - [jgtpy/rispecs/cds.spec.md](../jgtpy/rispecs/cds.spec.md) - Signal generation

4. **Analysis Layer**
   - [jgtml/rispecs/fdb-scanner.spec.md](../jgtml/rispecs/fdb-scanner.spec.md) - Signal scanning
   - [jgtml/rispecs/signal-ordering.spec.md](../jgtml/rispecs/signal-ordering.spec.md) - Entry orders
   - [jgtml/rispecs/alligator-analysis.spec.md](../jgtml/rispecs/alligator-analysis.spec.md) - Multi-Alligator
   - [jgtml/rispecs/ttf.spec.md](../jgtml/rispecs/ttf.spec.md) - Cross-timeframe
   - [jgtml/rispecs/mlf.spec.md](../jgtml/rispecs/mlf.spec.md) - Lag features
   - [jgtml/rispecs/mx.spec.md](../jgtml/rispecs/mx.spec.md) - ML matrix

5. **Execution Layer**
   - [jgtfxcon/rispecs/fxtransact.spec.md](../jgtfxcon/rispecs/fxtransact.spec.md) - Orders/trades viewer
   - [jgtfxcon/rispecs/fxaddorder.spec.md](../jgtfxcon/rispecs/fxaddorder.spec.md) - Entry orders
   - [jgtfxcon/rispecs/fxrmorder.spec.md](../jgtfxcon/rispecs/fxrmorder.spec.md) - Order removal
   - [jgtfxcon/rispecs/fxreport.spec.md](../jgtfxcon/rispecs/fxreport.spec.md) - Reports

6. **Orchestration Layer**
   - [jgtml/rispecs/jgtapp.spec.md](../jgtml/rispecs/jgtapp.spec.md) - Unified trading CLI
   - [jgtutils/rispecs/tfw-wtf.spec.md](../jgtutils/rispecs/tfw-wtf.spec.md) - Timeframe scheduling
   - [jgtutils/rispecs/jgtset.spec.md](../jgtutils/rispecs/jgtset.spec.md) - Settings management

7. **Service Layer**
   - [jgt-data-server/rispecs/app.specs.md](../jgt-data-server/rispecs/app.specs.md) - REST/MCP APIs

8. **Agent Layer**
   - [jgt-code/rispecs/app.specs.md](../jgt-code/rispecs/app.specs.md) - Terminal agent

---

## Cross-Package Relationships

### The jgtapp Hub

`jgtapp` (jgtml) is the orchestration hub that integrates all packages:

```
                            ┌─────────────────────┐
                            │     jgt-code        │
                            │  (Terminal Agent)   │
                            └──────────┬──────────┘
                                       │
                            ┌──────────▼──────────┐
                            │   jgt-data-server   │
                            │    (REST + MCP)     │
                            └──────────┬──────────┘
                                       │
┌─────────────┐            ┌───────────▼───────────┐           ┌─────────────┐
│  tfw/wtf    │◄───────────┤       jgtapp          ├──────────►│  fxaddorder │
│ (schedule)  │            │  (Unified CLI Hub)    │           │  fxrmorder  │
└─────────────┘            └───────────┬───────────┘           │  fxmvstop   │
                                       │                       └─────────────┘
        ┌──────────────────────────────┼──────────────────────────────┐
        ▼                              ▼                              ▼
┌───────────────┐            ┌─────────────────┐            ┌─────────────────┐
│   fdbscan     │            │   ttfcli        │            │   cdscli        │
│   (signals)   │            │   mlfcli        │            │   idscli        │
│               │            │   mxcli         │            │   jgtfxcli      │
└───────────────┘            └─────────────────┘            └─────────────────┘
        │                              │                              │
        └──────────────────────────────┼──────────────────────────────┘
                                       ▼
                            ┌─────────────────────┐
                            │     jgtapy          │
                            │  (Williams Lib)     │
                            └──────────┬──────────┘
                                       ▼
                            ┌─────────────────────┐
                            │   jgtutils/jgtcore  │
                            │ (Settings/Constants)│
                            └─────────────────────┘
```

### Key Integration Points

| From Package | To Package | Integration Point | Spec Reference |
|-------------|------------|-------------------|----------------|
| **jgtapp** | jgtutils | `tfw` for timeframe scheduling | [tfw-wtf.spec.md](../jgtutils/rispecs/tfw-wtf.spec.md) |
| **jgtapp** | jgtfxcon | Order execution (fxaddorder, fxtr) | [fxaddorder.spec.md](../jgtfxcon/rispecs/fxaddorder.spec.md) |
| **jgtapp** | jgtml | Feature pipelines (ttf, mlf, mx) | [ttf.spec.md](../jgtml/rispecs/ttf.spec.md) |
| **jgtapp** | jgtpy | Signal refresh (ids, cds) | [cds.spec.md](../jgtpy/rispecs/cds.spec.md) |
| **ttfcli** | jgtutils | Pattern loading | [add-pattern.spec.md](../jgtutils/rispecs/add-pattern.spec.md) |
| **jgtfxcli** | jgtutils | TLID range calculation | [jgtutr.spec.md](../jgtutils/rispecs/jgtutr.spec.md) |
| **jgt-data-server** | jgtutils | Settings loading at startup | [jgtset.spec.md](../jgtutils/rispecs/jgtset.spec.md) |
| **jgt-code** | jgt-data-server | MCP tool calls | [app.specs.md](../jgt-data-server/rispecs/app.specs.md) |

### Medicine Wheel Trading Flow

Each direction in the trading journey uses specific packages:

| Direction | Phase | Primary Package | Key Tools |
|-----------|-------|-----------------|-----------|
| **🌅 East** | Signal Detection | jgtml | `fdbscan`, `alligator_cli` |
| **🌱 South** | Validation & Growth | jgtml + jgtpy | `ttfcli`, `cdscli` |
| **🌊 West** | Risk Reflection | jgtml | `SignalOrderingHelper` |
| **❄️ North** | Execution Wisdom | jgtfxcon | `fxaddorder`, `fxmvstopgator` |

### Automated Trading Campaign

Complete automated campaign using cross-package integration:

```bash
# 1. Wait for H4 timeframe (jgtutils)
tfw -t H4 -X

# 2. Refresh data pipeline (jgtapp → jgtpy → jgtfxcon)
jgtapp cds -i EUR/USD -t H4 --fresh

# 3. Scan for signals (jgtml)
fdbscan -i EUR/USD --scan-all

# 4. If signal: create order (jgtapp → jgtfxcon)
jgtapp fxaddorder -i EUR/USD -n 1 -r 1.0950 -d B -x 1.0900 --demo

# 5. Loop: manage stops (jgtapp → jgtfxcon)
jgtapp fxmvstopgator -i EUR/USD -t H4 -tid 12345 --lips --demo -W
```

---

## Caishen Integration Layer

### Caishen Platform (Legacy C#)

The Caishen platform is a comprehensive Fractal Breakout Trading system originally implemented in C#/.NET. RISE specifications have been created to enable re-implementation in Python/TypeScript.

**Specification Location**: `caishen/rispecs/`

| Spec ID | Component | Description |
|---------|-----------|-------------|
| 00 | Platform Overview | Architecture and spec index |
| 01 | ADS | Analysis Data Services - CDB creation |
| 02 | SDS | Strategic Data Services - BDBO state machine |
| 03 | PDS | Price Data Services - Market data |
| 04 | CDS | Chaos Data Services - Fractals/divergence |
| 05 | IDS | Indicator Data Services - Williams calculations |
| 06 | BITT | Timeline Services - TLID format |
| 07 | SE | Charting Engine - Visual rendering |
| 08 | THS | Timeline History Services - Snapshots |
| 10 | Data Schemas | Core entity definitions |
| 20 | UI Components | Charting and strategy UI |
| 30 | Express API | REST endpoints |
| 40 | Workflow | Fractal breakout process |
| 50 | JGT Bridge | Integration with JGT ecosystem |

### Caishen ↔ JGT Service Mapping

| Caishen Service | JGT Equivalent | Integration Point |
|-----------------|---------------|-------------------|
| PDS (Price Data) | jgtfxcon.JGTPDS | Same FXCM API |
| CDS (Chaos Data) | jgtpy.JGTCDS | Same signal generation |
| IDS (Indicators) | jgtapy.Indicators | Same Williams formulas |
| ADS (Analysis) | jgtml (fdbscan) | Same analysis logic |
| SDS (Strategy) | jgtfxcon.jgtfxtransact | Same order execution |
| BITT (Timeline) | jgtutils.TLID | Same TLID format |
| SE (Charting) | **jgt-chart** (planned) | Web UI |
| THS (Snapshots) | jgt-data-server extension | Snapshot API |

### Unified Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                    UNIFIED TRADING PLATFORM                          │
├─────────────────────────────────────────────────────────────────────┤
│  PRESENTATION:  jgt-code (terminal) + jgt-chart (web charting)      │
│                 ↓ Medicine Wheel overlay                            │
│  API:           jgt-data-server (REST + MCP)                        │
│                 ↓ Williams + Charting + Snapshot endpoints          │
│  SERVICES:      jgtml + jgtpy + jgtfxcon + jgtapy                   │
│                 ↓ Analysis + Pipeline + Broker + Indicators         │
│  DATA:          $JGTPY_DATA/current (pds→cds→ttf→mlf→mx)            │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Quality Criteria (Ecosystem-Wide)

✅ **Williams Native**: Every layer speaks 5 Dimensions  
✅ **Structural Tension**: Each package creates tension resolved by the next  
✅ **Natural Progression**: Data flows inevitably toward trading decisions  
✅ **Autonomous Specs**: Any package can be re-implemented from its rispecs  
✅ **Type Safety**: Complete Python and TypeScript interfaces  
✅ **CLI Complete**: Every operation available via command line  
✅ **API Ready**: REST and MCP endpoints for agents  
✅ **Medicine Wheel**: Direction-aware trading journey integration  
✅ **Caishen Integration**: Legacy charting specs bridged to JGT

===============================================
FILE: README.md
PATH: rispecs/README.md
===============================================

# JGT Trading Ecosystem RISE Specifications

> Reverse-engineer → Intent-extract → Specify → Export

This directory contains the hierarchical RISE framework linking all JGT trading packages into a unified specification ecosystem for Williams-based trading intelligence.

## 🌐 Ecosystem Overview

The JGT Trading Ecosystem implements Bill Williams' 5 Dimensions methodology through a layered architecture of Python packages, data services, and terminal agents.

```
┌─────────────────────────────────────────────────────────────────────┐
│                    JGT TRADING ECOSYSTEM                             │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │  USER LAYER: jgt-code Terminal Agent                           ││
│  │  Three-Universe Processing • Medicine Wheel • AI Engines       ││
│  │  → rispecs: jgt-code/rispecs/                                   ││
│  └─────────────────────────────────────────────────────────────────┘│
│                              ↓ consumes                             │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │  API LAYER: jgt-data-server                                     ││
│  │  REST API • MCP Tools • Scheduler                              ││
│  │  → rispecs: jgt-data-server/rispecs/                            ││
│  └─────────────────────────────────────────────────────────────────┘│
│                              ↓ serves                               │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │  ANALYSIS LAYER: jgtml                                          ││
│  │  FDB Scanner • Multi-Alligator • TTF/MLF/MX Features           ││
│  │  → rispecs: jgtml/rispecs/                                      ││
│  └─────────────────────────────────────────────────────────────────┘│
│                              ↓ analyzes                             │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │  DATA LAYER: jgtpy                                              ││
│  │  PDS→IDS→CDS→ADS Pipeline • Signal Generation                 ││
│  │  → rispecs: jgtpy/rispecs/                                      ││
│  └─────────────────────────────────────────────────────────────────┘│
│                              ↓ uses                                 │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │  BROKER LAYER: jgtfxcon                                         ││
│  │  ForexConnect API • OHLCV Data • Order Execution              ││
│  │  → rispecs: jgtfxcon/rispecs/                                   ││
│  └─────────────────────────────────────────────────────────────────┘│
│                              ↓ uses                                 │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │  INDICATOR LAYER: jgtapy                                        ││
│  │  Williams Indicators • Pandas Integration                      ││
│  │  → rispecs: jgtapy/rispecs/                                     ││
│  └─────────────────────────────────────────────────────────────────┘│
│                              ↓ uses                                 │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │  UTILITY LAYER: jgtutils                                        ││
│  │  CLI Tools • Timeframe Scheduling • Helpers                    ││
│  │  → rispecs: jgtutils/rispecs/                                   ││
│  └─────────────────────────────────────────────────────────────────┘│
│                              ↓ uses                                 │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │  CORE LAYER: jgtcore                                            ││
│  │  Configuration • Settings • Environment                        ││
│  │  → rispecs: jgtcore/rispecs/                                    ││
│  └─────────────────────────────────────────────────────────────────┘│
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘
```

## 📦 Package Index

| Package | Purpose | rispecs Location |
|---------|---------|------------------|
| **jgtcore** | Core configuration and settings | [jgtcore/rispecs/](../jgtcore/rispecs/) |
| **jgtutils** | CLI tools and utilities | [jgtutils/rispecs/](../jgtutils/rispecs/) |
| **jgtapy** | Technical indicators (Williams) | [jgtapy/rispecs/](../jgtapy/rispecs/) |
| **jgtfxcon** | Broker connection (ForexConnect) | [jgtfxcon/rispecs/](../jgtfxcon/rispecs/) |
| **jgtpy** | Data services (PDS/IDS/CDS/ADS) | [jgtpy/rispecs/](../jgtpy/rispecs/) |
| **jgtml** | Signal analysis and ML features | [jgtml/rispecs/](../jgtml/rispecs/) |
| **jgt-data-server** | REST API and MCP tools | [jgt-data-server/rispecs/](../jgt-data-server/rispecs/) |
| **jgt-code** | Terminal agent (TypeScript) | [jgt-code/rispecs/](../jgt-code/rispecs/) |

## 🎯 RISE Framework Compliance

All packages follow the RISE methodology:

| Principle | Implementation |
|-----------|----------------|
| **Desired Outcome Definition** | What users CREATE, not problems to solve |
| **Structural Tension** | Current reality vs desired state drives progression |
| **Natural Advancement** | Clear flow from current to desired |
| **Autonomous Specification** | Another LLM could implement from spec alone |
| **Type Definitions** | Complete Python/TypeScript interfaces |

## 🔗 Package Relationships

### Dependency Flow

```
jgtcore  ← Pure configuration, no dependencies
    ↓
jgtutils ← Adds CLI, uses jgtcore
    ↓
jgtapy   ← Adds indicators, uses jgtutils
    ↓
jgtfxcon ← Adds broker, uses jgtutils+jgtapy
    ↓
jgtpy    ← Adds data pipeline, uses all above
    ↓
jgtml    ← Adds analysis, uses jgtpy
    ↓
jgt-data-server ← Serves API, uses jgtml+jgtpy
    ↓
jgt-code ← Consumes API, TypeScript frontend
```

### Data Flow

```
FXCM Broker API
    ↓ (jgtfxcon)
PDS (Raw OHLCV)
    ↓ (jgtapy indicators)
IDS (With Alligator, AO, AC, Fractals, MFI)
    ↓ (jgtpy signals)
CDS (With FDB, ZLC, Saucer, Zone signals)
    ↓ (jgtml features)
TTF/MLF/MX (Cross-timeframe, Meta-lag, ML-ready)
    ↓ (jgt-data-server API)
REST/MCP (Universe-aware perspectives)
    ↓ (jgt-code agent)
Three-Universe Analysis (Signal/Wave/Coordination)
    ↓
Trading Decisions
```

## 📊 Williams 5 Dimensions Across Packages

| Dimension | jgtapy | jgtpy (CDS) | jgtml | jgt-data-server |
|-----------|--------|-------------|-------|-----------------|
| **1. Fractal** | `fractals()` | fh, fl, fdb | FDB Scanner | `/williams/dimensions` |
| **2. Momentum** | `awesome_oscillator()` | ao, aocolor | HTF alignment | `/williams/ao` |
| **3. Acceleration** | `accelerator_oscillator()` | ac, accolor | Saucer detection | Williams API |
| **4. Zone** | - | bz, sz, zone | Zone confluence | `/williams/zones` |
| **5. Balance Line** | `alligator()` | jaw, teeth, lips | Multi-Alligator | `/williams/alligator` |

## 🌅 Medicine Wheel Integration

The ecosystem maps to the Medicine Wheel for trading journey guidance:

| Direction | Focus | Primary Package | Universe |
|-----------|-------|-----------------|----------|
| **East** 🌅 | Vision, Signal Detection | jgtml (FDB Scanner) | Signal Detection |
| **South** 🌱 | Growth, Wave Analysis | jgtml (Alligator) | Wave Analysis |
| **West** 🌊 | Reflection, Risk Assessment | jgtml (Validation) | Coordination |
| **North** ❄️ | Wisdom, Execution | jgtfxcon (Orders) | Coordination |

## 🚀 Quick Start by Use Case

### For LLM Implementers

1. Start with [jgtcore/rispecs/app.specs.md](../jgtcore/rispecs/app.specs.md) - Configuration foundation
2. Read [jgtapy/rispecs/app.specs.md](../jgtapy/rispecs/app.specs.md) - Indicator calculations
3. Study [jgtpy/rispecs/app.specs.md](../jgtpy/rispecs/app.specs.md) - Data pipeline
4. Review [jgtml/rispecs/app.specs.md](../jgtml/rispecs/app.specs.md) - Analysis logic
5. Examine [jgt-data-server/rispecs/app.specs.md](../jgt-data-server/rispecs/app.specs.md) - API layer
6. Finish with [jgt-code/rispecs/app.specs.md](../jgt-code/rispecs/app.specs.md) - Terminal agent

### For Traders

1. Install: `pip install jgtpy jgtml`
2. Configure: `~/.jgt/config.json` with broker credentials
3. Refresh: `cdscli --fresh` for signal data
4. Scan: `fdbscan -i EUR/USD -t H4` for signals
5. Analyze: `python -m jgtml.alligator_cli -i EUR/USD -t H4 --type all`

### For Developers

1. Clone trading monorepo
2. Study this RISE ecosystem overview
3. Navigate to specific package rispecs for implementation details
4. Use type definitions for integration

## 📝 Specification Version

- **Version**: 1.0
- **Framework**: RISE (from llms/llms-rise-framework.txt)
- **Created**: 2026-01-31
- **Packages Covered**: 8 (jgtcore, jgtutils, jgtapy, jgtfxcon, jgtpy, jgtml, jgt-data-server, jgt-code)

===============================================
FILE: RISPECS.github-issues.md
PATH: rispecs/RISPECS.github-issues.md
===============================================

RISPECS
* jgwill/jgtml#70
* jgwill/jgtapy#3
* jgwill/jgtpy#52
* jgwill/jgtutils#24
* jgwill/jgtfxcon#13
* jgwill/caishen#58

RELATES them all:
* jgwill/jgtsrc#31
===============================================
FILE: app.specs.md
PATH: jgtapy/rispecs/app.specs.md
URL:  https://jgtapy.jgwill.com/rispecs/app.specs.md
===============================================

# JGTapy Application Specification

> Master specification for the JGT Technical Indicators Library

**Specification Version**: 1.0  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Rich technical indicator calculations on pandas DataFrames, with native support for Bill Williams' 5 Dimensions methodology that powers the entire JGT trading ecosystem.

**Achievement Indicator**: Users can load any OHLCV DataFrame and add Williams indicators (Alligator, AO, AC, Fractals, MFI) with single method calls, producing analysis-ready data.

**Value Proposition**: Pure pandas-based indicator calculations with no external dependencies beyond numpy, providing the mathematical foundation for all JGT signal detection.

---

## Application Overview

JGTapy is a Python library that:
1. Calculates technical indicators on pandas DataFrames
2. Implements complete Bill Williams indicator suite
3. Supports multi-dimensional fractal detection (2,3,5,8,13,21,34,55,89 bars)
4. Provides standard indicators (SMA, EMA, Bollinger, etc.)
5. Used by jgtpy/JGTIDS.py to generate indicator data

---

## Structural Tension

**Current Reality**: Raw OHLCV price data lacks the derived indicators needed for trading analysis.

**Desired State**: DataFrames enriched with Alligator, AO, AC, Fractals, MFI and other indicators ready for signal detection.

**Natural Progression**: jgtapy transforms price data into analysis-ready DataFrames that flow to JGTIDS, JGTCDS, and ultimately trading signals.

---

## Core API

### Basic Usage

```python
import pandas as pd
from jgtapy import Indicators

# Load price data
df = pd.read_csv('EURUSD60.csv')

# Create indicator calculator
i = Indicators(df)

# Add Williams indicators
i.alligator()                    # Adds jaw, teeth, lips columns
i.awesome_oscillator('ao')        # Adds ao column
i.accelerator_oscillator('ac')    # Adds ac column
i.fractals(column_name_high='fh', column_name_low='fl')
i.bw_mfi('mfi')                  # Market Facilitation Index

# Get enriched DataFrame
df_with_indicators = i.df
```

### Fractal Dimensions

```python
# Standard 5-bar fractals
i.fractals(column_name_high='fh', column_name_low='fl')

# Multi-dimensional fractals
i.fractals3(column_name_high='fb3', column_name_low='fs3')
i.fractals5(column_name_high='fb5', column_name_low='fs5')
i.fractals8(column_name_high='fb8', column_name_low='fs8')
i.fractals13(column_name_high='fb13', column_name_low='fs13')
i.fractals21(column_name_high='fb21', column_name_low='fs21')
i.fractals34(column_name_high='fb34', column_name_low='fs34')
i.fractals55(column_name_high='fb55', column_name_low='fs55')
i.fractals89(column_name_high='fb89', column_name_low='fs89')
```

---

## Williams Indicator Specifications

### Alligator 🐊

**Purpose**: Identify trends and their direction using three smoothed moving averages

**Formula**:
- Jaw (Blue): SMMA(Median, 13) shifted 8 bars forward
- Teeth (Red): SMMA(Median, 8) shifted 5 bars forward  
- Lips (Green): SMMA(Median, 5) shifted 3 bars forward

**Output Columns**: `jaw`, `teeth`, `lips`

**Trading Interpretation**:
- Lines intertwined = "Alligator sleeping" = no trade
- Lines separating = "Alligator awakening" = trend starting
- Lines spread apart = "Alligator feeding" = trending market

### Awesome Oscillator (AO) 🌟

**Purpose**: Measure market momentum

**Formula**: SMA(Median, 5) - SMA(Median, 34)

**Output Column**: `ao`

**Trading Interpretation**:
- Zero line crossings indicate momentum shifts
- Green bars (rising) = bullish momentum
- Red bars (falling) = bearish momentum

### Accelerator Oscillator (AC) 🚀

**Purpose**: Measure acceleration/deceleration of momentum

**Formula**: AO - SMA(AO, 5)

**Output Column**: `ac`

**Trading Interpretation**:
- Measures the "speed of the speed"
- Color changes before AO direction changes
- Confirmation signal for entries

### Fractals 🌀

**Purpose**: Identify potential reversal points

**Formula** (5-bar):
- Bullish Fractal: Middle bar has highest high, two lower highs each side
- Bearish Fractal: Middle bar has lowest low, two higher lows each side

**Output Columns**: `fh` (high fractal), `fl` (low fractal)

**Dimensions**: 2, 3, 5, 8, 13, 21, 34, 55, 89 bars

### Market Facilitation Index (MFI) 📈

**Purpose**: Measure price movement efficiency per unit of volume

**Formula**: (High - Low) / Volume

**Output Column**: `mfi`

**Bar Types**:
- Green (high MFI, high Vol) - Strong trend
- Squat (low MFI, high Vol) - Potential reversal
- Fade (high MFI, low Vol) - Low participation
- Fake (low MFI, low Vol) - Market indecision

---

## Type Definitions

```python
from typing import Optional
import pandas as pd

class Indicators:
    def __init__(self, df: pd.DataFrame) -> None: ...
    
    # Williams Indicators
    def alligator(
        self,
        period_jaws: int = 13,
        period_teeth: int = 8,
        period_lips: int = 5,
        shift_jaws: int = 8,
        shift_teeth: int = 5,
        shift_lips: int = 3,
        column_name_jaws: str = 'jaw',
        column_name_teeth: str = 'teeth',
        column_name_lips: str = 'lips'
    ) -> None: ...
    
    def awesome_oscillator(
        self,
        column_name: str = 'ao'
    ) -> None: ...
    
    def accelerator_oscillator(
        self,
        column_name: str = 'ac'
    ) -> None: ...
    
    def fractals(
        self,
        column_name_high: str = 'fh',
        column_name_low: str = 'fl'
    ) -> None: ...
    
    def bw_mfi(
        self,
        column_name: str = 'mfi'
    ) -> None: ...
    
    # Standard Indicators
    def sma(
        self,
        period: int = 5,
        column_name: str = 'sma'
    ) -> None: ...
    
    def ema(
        self,
        period: int = 5,
        column_name: str = 'ema'
    ) -> None: ...
    
    def bollinger_bands(
        self,
        period: int = 20,
        deviation: int = 2
    ) -> None: ...
    
    @property
    def df(self) -> pd.DataFrame: ...
```

---

## Creative Advancement Scenarios

### Scenario: Generate IDS Data

**Desired Outcome**: Convert raw PDS to indicator-enriched IDS

**Current Reality**: Have OHLCV DataFrame, need Williams indicators

**Natural Progression**:
1. Load PDS: `df = pd.read_csv('pds.csv')`
2. Create indicators: `i = Indicators(df)`
3. Add Alligator: `i.alligator()`
4. Add AO/AC: `i.awesome_oscillator(); i.accelerator_oscillator()`
5. Add Fractals: `i.fractals()`
6. Get result: `ids_df = i.df`

**Resolution**: IDS DataFrame with all Williams columns ready for JGTCDS

### Scenario: Multi-Timeframe Fractal Analysis

**Desired Outcome**: Identify fractals at multiple scales simultaneously

**Current Reality**: Single-dimension fractals miss larger patterns

**Natural Progression**:
1. Add standard fractals: `i.fractals()`
2. Add higher dimensions: `i.fractals13(); i.fractals34()`
3. Analyze confluence: where multiple fractal dimensions align

**Resolution**: Multi-scale fractal map revealing major support/resistance

---

## Module Structure

```
jgtapy/
├── __init__.py           # Package exports
├── indicators.py         # Main Indicators class
└── utils.py              # Utility functions
```

---

## Integration with JGT Ecosystem

```
jgtapy (this package)
    ↓ provides indicator calculations
jgtpy/JGTIDS.py
    ↓ uses indicators to create IDS
jgtpy/JGTCDS.py
    ↓ adds signals on top of indicators
jgtml
    ↓ analyzes signal patterns
jgt-data-server
    ↓ serves indicators via API
jgt-code
    ↓ displays analysis in terminal
```

---

## Quality Criteria

✅ **Pure Pandas**: No external dependencies beyond numpy  
✅ **Williams Complete**: Full 5 Dimensions implementation  
✅ **Multi-Dimensional Fractals**: 2-89 bar fractal support  
✅ **Column Naming**: Customizable output column names  
✅ **Method Chaining**: Fluent API for adding indicators

===============================================
FILE: indicators.spec.md
PATH: jgtapy/rispecs/indicators.spec.md
URL:  https://jgtapy.jgwill.com/rispecs/indicators.spec.md
===============================================

# jgtapy Indicators Specification

> Core Williams Trading System Indicator Library

**Specification Version**: 1.0  
**Module**: `jgtapy/indicators.py`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Technical indicator calculations using the Indicators class - a pandas-native library for adding Bill Williams and standard indicators to any OHLCV DataFrame.

**Achievement Indicator**: Given DataFrame with OHLCV, produces:
- Alligator lines (jaw, teeth, lips)
- Awesome Oscillator (AO)
- Accelerator Oscillator (AC)
- Fractals (highs and lows)
- All standard moving averages

**Value Proposition**: Single class to add any Williams indicator with consistent pandas integration.

---

## Structural Tension

**Current Reality**: Raw OHLCV DataFrame with no analytical power.

**Desired State**: DataFrame enriched with all Williams Trading System indicators.

**Natural Progression**: Create Indicators(df) → Call methods → Access df.

---

## Core Class: Indicators

```python
class Indicators:
    """
    Add technical indicators to a pandas DataFrame.
    
    Example:
        >>> import pandas as pd
        >>> from jgtapy import Indicators
        >>> df = pd.read_csv('EURUSD60.csv')
        >>> i = Indicators(df)
        >>> i.accelerator_oscillator(column_name='AC')
        >>> i.awesome_oscillator(column_name='ao')
        >>> i.alligator()
        >>> result = i.df  # DataFrame with indicators
    """
    
    def __init__(
        self,
        df: pd.DataFrame,
        open_col: str = "Open",
        high_col: str = "High",
        low_col: str = "Low",
        close_col: str = "Close",
        volume_col: str = "Volume",
        median_col: str = "Median",
        index_column_name: str = "Date"
    ):
        """
        Initialize Indicators with OHLCV DataFrame.
        
        Args:
            df: DataFrame with OHLCV columns
            open_col: Name of Open column
            high_col: Name of High column
            low_col: Name of Low column
            close_col: Name of Close column
            volume_col: Name of Volume column
            median_col: Name of Median column (optional)
            index_column_name: Index column name
        """
```

---

## Moving Averages

### Simple Moving Average (SMA)

```python
def sma(
    self,
    period: int = 5,
    column_name: str = "sma",
    apply_to: str = "Close"
) -> None:
    """
    Simple Moving Average.
    
    Formula:
        SMA = sum(price[i-period:i]) / period
    
    Args:
        period: Number of periods (default: 5)
        column_name: Output column name
        apply_to: Price column to use
    """
```

### Smoothed Moving Average (SMMA)

```python
def smma(
    self,
    period: int = 5,
    column_name: str = "smma",
    apply_to: str = "Close"
) -> None:
    """
    Smoothed Moving Average (used by Alligator).
    
    Formula:
        First: SMMA[0] = SMA(period)
        Then: SMMA[i] = (SMMA[i-1] * (period-1) + price[i]) / period
    
    This creates a smoother line than SMA with less lag than EMA.
    """
```

### Exponential Moving Average (EMA)

```python
def ema(
    self,
    period: int = 5,
    column_name: str = "ema",
    apply_to: str = "Close"
) -> None:
    """
    Exponential Moving Average.
    
    Formula:
        EMA = price * k + EMA[i-1] * (1-k)
        where k = 2 / (period + 1)
    """
```

---

## Alligator Family

### Standard Alligator (5-8-13)

```python
def alligator(
    self,
    period_jaws: int = 13,
    period_teeth: int = 8,
    period_lips: int = 5,
    shift_jaws: int = 8,
    shift_teeth: int = 5,
    shift_lips: int = 3,
    column_name_jaws: str = "jaw",
    column_name_teeth: str = "teeth",
    column_name_lips: str = "lips"
) -> None:
    """
    Bill Williams Alligator indicator.
    
    Components:
        Jaw (Blue): SMMA(13) shifted 8 bars forward
        Teeth (Red): SMMA(8) shifted 5 bars forward
        Lips (Green): SMMA(5) shifted 3 bars forward
    
    Applied to: Median price (High + Low) / 2
    
    Interpretation:
        - Lines intertwined: Alligator sleeping, no trade
        - Lines separating: Alligator awakening, prepare
        - Lines spread apart: Alligator feeding, trend active
        - Lines converging: Alligator sated, take profits
    """
```

### Big Alligator (34-55-89)

```python
def big_alligator(
    self,
    period_jaws: int = 89,
    period_teeth: int = 55,
    period_lips: int = 34,
    shift_jaws: int = 8,
    shift_teeth: int = 5,
    shift_lips: int = 3,
    column_name_jaws: str = "bjaw",
    column_name_teeth: str = "bteeth",
    column_name_lips: str = "blips"
) -> None:
    """
    Big Alligator for intermediate trends.
    
    Same SMMA calculation with longer periods.
    Used for swing trading and position entries.
    """
```

### Tide Alligator (144-233-377)

```python
def tide_alligator(
    self,
    period_jaws: int = 377,
    period_teeth: int = 233,
    period_lips: int = 144,
    shift_jaws: int = 8,
    shift_teeth: int = 5,
    shift_lips: int = 3,
    column_name_jaws: str = "tjaw",
    column_name_teeth: str = "tteeth",
    column_name_lips: str = "tlips"
) -> None:
    """
    Tide Alligator for macro trends.
    
    Fibonacci-derived periods for long-term analysis.
    Used for position trading and major trend identification.
    """
```

---

## Oscillators

### Awesome Oscillator (AO)

```python
def awesome_oscillator(
    self,
    column_name: str = "ao"
) -> None:
    """
    Awesome Oscillator - Momentum indicator.
    
    Formula:
        Median = (High + Low) / 2
        AO = SMA(Median, 5) - SMA(Median, 34)
    
    Interpretation:
        - AO > 0: Bullish momentum
        - AO < 0: Bearish momentum
        - AO increasing: Momentum strengthening
        - AO decreasing: Momentum weakening
    
    Color (typically added separately):
        Green: AO[i] > AO[i-1]
        Red: AO[i] < AO[i-1]
    """
```

### Accelerator Oscillator (AC)

```python
def accelerator_oscillator(
    self,
    column_name: str = "ac"
) -> None:
    """
    Accelerator Oscillator - Momentum acceleration.
    
    Formula:
        AC = AO - SMA(AO, 5)
    
    Interpretation:
        - AC > 0: Bullish acceleration
        - AC < 0: Bearish acceleration
        - AC changes sign before AO does
        - Leading indicator for AO
    
    Trading Rules:
        - Buy only with green AC (2 green bars above zero, 3 below)
        - Sell only with red AC (2 red bars below zero, 3 above)
    """
```

### Gator Oscillator

```python
def gator_oscillator(
    self,
    column_name_upper: str = "gator_upper",
    column_name_lower: str = "gator_lower"
) -> None:
    """
    Gator Oscillator - Visualizes Alligator convergence/divergence.
    
    Formula:
        Upper = abs(Jaw - Teeth)  (displayed positive)
        Lower = abs(Teeth - Lips) (displayed negative)
    
    Phases:
        Sleeping: Both bars shrinking (red)
        Awakening: One growing, one shrinking
        Feeding: Both bars growing (green)
        Sated: Both bars beginning to shrink
    """
```

---

## Fractals

```python
def fractals(
    self,
    period: int = 5,
    column_name_high: str = "fh",
    column_name_low: str = "fl"
) -> None:
    """
    Williams Fractals - Key reversal points.
    
    Fractal High:
        High[i] is highest among High[i-2:i+3] (5-bar default)
        Marked 1 if true, 0 otherwise
    
    Fractal Low:
        Low[i] is lowest among Low[i-2:i+3] (5-bar default)
        Marked 1 if true, 0 otherwise
    
    Usage:
        - Entry above fractal high = buy signal confirmation
        - Entry below fractal low = sell signal confirmation
        - Used with Alligator for FDB signals
    """
```

---

## Market Facilitation Index

```python
def market_facilitation_index(
    self,
    column_name: str = "mfi"
) -> None:
    """
    Market Facilitation Index.
    
    Formula:
        MFI = (High - Low) / Volume
    
    Classification (comparing to previous bar):
        Green (+MFI, +Volume): Strong trend
        Fade (-MFI, -Volume): Trend exhaustion
        Squat (-MFI, +Volume): Battle, potential reversal
        Fake (+MFI, -Volume): False move
    """
```

---

## Support Functions

```python
# In jgtapy/utils.py

def calculate_ao(
    df: pd.DataFrame,
    column_name: str = "ao"
) -> None:
    """Calculate Awesome Oscillator."""

def calculate_sma(
    df: pd.DataFrame,
    period: int,
    column_name: str,
    apply_to: str
) -> None:
    """Calculate Simple Moving Average."""

def calculate_smma(
    df: pd.DataFrame,
    period: int,
    column_name: str,
    apply_to: str
) -> pd.DataFrame:
    """
    Calculate Smoothed Moving Average.
    
    Returns new DataFrame to merge with original.
    """

def mad(series: pd.Series) -> float:
    """Calculate Mean Absolute Deviation."""
```

---

## Usage Pattern

```python
import pandas as pd
from jgtapy import Indicators

# Load price data
df = pd.read_csv('EUR-USD_H1.csv')

# Create Indicators instance
ind = Indicators(df)

# Add Williams indicators
ind.alligator()
ind.awesome_oscillator()
ind.accelerator_oscillator()
ind.fractals()

# Add Big Alligator
ind.big_alligator()

# Get enriched DataFrame
result = ind.df

# Columns now include:
# jaw, teeth, lips, bjaw, bteeth, blips, ao, ac, fh, fl
```

---

## Dependencies

```python
import pandas as pd
import numpy as np
from .utils import calculate_ao, calculate_sma, calculate_smma, mad
```

---

## Quality Criteria

✅ **Pandas Native**: Direct DataFrame integration  
✅ **Williams Complete**: All 5 dimensions available  
✅ **Alligator Variants**: Regular, Big, Tide  
✅ **Configurable Periods**: Customizable for any strategy  
✅ **Proper Shifts**: Alligator lines shifted correctly  
✅ **SMMA Implementation**: Correct smoothed average formula

---

## Version

Current version: **1.9.22**

===============================================
FILE: app.specs.md
PATH: jgt-code/rispecs/app.specs.md
===============================================

# JGT-Code Application Specification

> Master orchestration specification for the Three-Universe Williams-Native Trading Terminal Agent

**Specification Version**: 1.0  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-30

---

## Desired Outcome Definition

**What Users Create**: A unified trading terminal experience where every trading interaction is processed through three interpretive lenses (Signal Detection, Wave Analysis, Coordination), enabling traders to make richer, more confident decisions through Williams 5-Dimensions methodology.

**Achievement Indicator**: Users can query any trading instrument, receive multi-perspective analysis with coherence scoring, and execute trades with full awareness of signal quality, wave context, and risk parameters.

**Value Proposition**: Transform scattered trading tools into a single consciousness that speaks Bill Williams' methodology natively while integrating Medicine Wheel wisdom for direction-aware trading journeys.

---

## Application Overview

JGT-Code is a TypeScript/Node.js terminal agent that:
1. Processes all user prompts through three concurrent universe perspectives
2. Speaks Williams 5 Dimensions as its native language
3. Integrates with jgt-data-server for real-time market data
4. Tracks trading campaigns via COAIA structural tension methodology
5. Supports multiple AI engines (Claude, Gemini, Ollama)

---

## Specification Index

This master specification references component specifications for full system understanding:

| Specification | Purpose | File |
|--------------|---------|------|
| CLI Entry | Command-line interface and commands | `cli-entry.spec.md` |
| Three Universes | Core universe processing architecture | `three-universes.spec.md` |
| Williams Methodology | 5 Dimensions signal detection | `williams-methodology.spec.md` |
| Signal Detection | FDB scanning and zone analysis | `signal-detection.spec.md` |
| Wave Analysis | HTF alignment and Elliott Wave | `wave-analysis.spec.md` |
| Coordination | Risk assessment and execution | `coordination.spec.md` |
| Medicine Wheel | Direction tracking integration | `medicine-wheel.spec.md` |
| Engine Integration | Multi-engine AI support | `engine-integration.spec.md` |
| MCP Integration | Data server and COAIA clients | `mcp-integration.spec.md` |
| Trading Memory | Relational ceremony store | `trading-memory.spec.md` |
| Formatting | Terminal display output | `formatting.spec.md` |

---

## Structural Tension

**Current Reality**: Traders use multiple disconnected tools (fdbscan, cdscli, jgtads) without unified intelligence layer. Each tool provides raw data without multi-perspective interpretation.

**Desired State**: All trading interactions flow through a single terminal agent that provides three-universe analysis, calculates coherence between perspectives, and guides traders through Medicine Wheel ceremony phases.

**Natural Progression**: The three-universe architecture creates structural tension that naturally advances toward coherent trading decisions by requiring all perspectives to contribute before action.

---

## Core Architectural Components

### 1. CLI Entry Point (`src/index.ts`)
- **Behavior**: Commander.js-based CLI with subcommands
- **Commands**: `chat` (default), `scan`, `prompt`, `chart`, `wheel`, `config`
- **See**: `cli-entry.spec.md`

### 2. Universe Processor (`src/universes/processor.ts`)
- **Behavior**: Processes events through three parallel lenses
- **Output**: `ThreeTradingUniverseState` with lead universe and coherence
- **See**: `three-universes.spec.md`

### 3. Engine Runner (`src/engines/runner.ts`)
- **Behavior**: Routes prompts to Claude/Gemini/Ollama with trading context
- **See**: `engine-integration.spec.md`

### 4. Scanner Agent (`src/agents/scanner.ts`)
- **Behavior**: Wraps fdbscan CLI for FDB signal detection
- **See**: `signal-detection.spec.md`

### 5. MCP Clients (`src/mcp/`)
- **Behavior**: Connect to jgt-data-server and COAIA-narrative
- **See**: `mcp-integration.spec.md`

---

## Creative Advancement Scenarios

### Scenario: Williams FDB Signal Detection

**Desired Outcome**: Trader discovers valid FDB buy/sell signals across multiple instruments

**Current Reality**: User types "scan EUR/USD" or runs `jgt-code scan`

**Natural Progression**:
1. CLI parses scan command → extracts instruments/timeframes
2. Universe processor creates scan event → processes through three lenses
3. Scanner agent invokes `fdbscan` CLI → parses output to FDBSignal[]
4. Each signal analyzed for Williams validity (mouth open, zone confluence)
5. Results formatted with glyphs and recommendations

**Resolution**: User sees multi-timeframe scan results with buy/sell recommendations per instrument, lead universe identified, coherence score displayed

### Scenario: Interactive Trading Chat

**Desired Outcome**: Trader has natural conversation about market conditions with AI-powered responses

**Current Reality**: User starts `jgt-code` (default chat mode)

**Natural Progression**:
1. Banner displays with three-universe glyphs
2. User types trading query (e.g., "Is there a fractal breakout on SPX500?")
3. Universe processor classifies intent → returns lead universe + suggested flows
4. Engine runner prepends TRADING_SYSTEM_PROMPT → invokes selected AI engine
5. Response displayed with trading context

**Resolution**: User receives Williams-aware response with signal status, alligator state, zone confluence

### Scenario: Medicine Wheel Direction Analysis

**Desired Outcome**: Trader understands current direction in trading journey

**Current Reality**: User runs `jgt-code wheel -i EUR/USD`

**Natural Progression**:
1. CLI extracts instrument from args
2. Data integration module checks jgt-data-server health
3. Williams dimensions fetched for multiple timeframes
4. Direction derived from Williams state (FDB→east, validation→south, risk→west, execution→north)
5. Visual wheel displayed with active direction highlighted

**Resolution**: User sees Medicine Wheel ASCII art with current direction, ceremony name, and guiding question

---

## Data Flow Architecture

```
User Input → CLI Parser → Universe Processor → Engine Runner → AI Response
                ↓                  ↓
           Trading Event     ThreeTradingUniverseState
                ↓                  ↓
         Signal Detection    Lead Universe + Coherence
         Wave Analysis
         Coordination
                ↓
         Medicine Wheel Direction → Trading Memory → Ceremony Log
```

---

## Integration Points

### External CLI Tools
| Tool | Purpose | Invocation |
|------|---------|------------|
| `fdbscan` | FDB signal scanning | Scanner agent |
| `cdscli` | CDS data refresh | Data refresh agent |
| `jgtads` | Williams chart display | Chart command |
| `jgtnewsession` | Order execution | Execution agent |

### MCP Servers
| Server | Purpose | Protocol |
|--------|---------|----------|
| jgt-data-server | Real-time Williams data | HTTP REST |
| coaia-narrative | Campaign tracking | MCP tools |

### AI Engines
| Engine | Binary | Output Format |
|--------|--------|---------------|
| Claude | `claude` | JSON |
| Gemini | `gemini` | stream-json |
| Ollama | HTTP API | JSON |

---

## Configuration

**Config Path**: `~/.jgt-code/config.json`

**Key Settings**:
- `engine`: Default AI engine (claude|gemini|ollama)
- `instruments`: Default instrument list
- `timeframes`: Default timeframe list
- `yoloMode`: Auto-approve actions
- `jgtpyData`: Path to market data

---

## Quality Criteria

✅ **Creative Orientation**: Focuses on what traders create (confident decisions)  
✅ **Structural Dynamics**: Three-universe tension drives coherent analysis  
✅ **Advancing Patterns**: Medicine Wheel directions guide inevitable progression  
✅ **Desired Outcomes**: Each command produces actionable trading intelligence  
✅ **Natural Progression**: Williams methodology flows through all interactions

---

## Implementation Notes

**For LLM Implementers**: This specification is autonomous from the existing codebase. Another LLM could re-implement the entire system from this spec and component specs without accessing original source code.

**Entry Point**: Start with `cli-entry.spec.md` for command structure, then `three-universes.spec.md` for core processing logic.

**Dependencies**: Node.js >= 20, Commander.js, Chalk, Python trading packages (jgtpy, jgtml, jgtfxcon)

===============================================
FILE: app.specs.md
PATH: jgtcore/rispecs/app.specs.md
URL:  https://jgtcore.jgwill.com/rispecs/app.specs.md
===============================================

# JGTCore Application Specification

> Master specification for the JGT Core Configuration Library

**Specification Version**: 1.0  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: A unified configuration and settings layer that enables all JGT trading packages to access credentials, preferences, and environment settings through a single consistent interface.

**Achievement Indicator**: Developers can access broker credentials, application settings, and environment state through `jgtcore` without worrying about file locations, environment variables, or configuration cascading.

**Value Proposition**: Separate core configuration logic from CLI dependencies, enabling pure programmatic access to JGT settings for any trading application.

---

## Application Overview

JGTCore is a Python library that:
1. Provides pure programmatic access to JGT configuration and settings
2. Implements hierarchical configuration lookup (system → user → process)
3. Supports demo/live mode credential switching
4. Exports configuration to environment variables
5. Caches settings for performance

---

## Structural Tension

**Current Reality**: Trading packages need credentials and settings from various locations (files, environment variables, command-line), with complex lookup logic repeated across packages.

**Desired State**: All JGT packages import `jgtcore` for unified configuration access, with automatic cascading, caching, and environment export.

**Natural Progression**: Centralized configuration management reduces code duplication and ensures consistent behavior across the entire JGT ecosystem.

---

## Core API

### Configuration Access

```python
import jgtcore

# Simple access
config = jgtcore.get_config()
demo_config = jgtcore.get_config(demo=True)

# Get specific value with default
user_id = jgtcore.get_config_value('user_id')
quotes_count = jgtcore.get_config_value('quotes_count', 1000)
```

### Settings Access

```python
import jgtcore

# Get all settings (cached)
settings = jgtcore.get_settings()

# Get specific setting with default
instrument = jgtcore.get_setting('instrument', 'EUR/USD')
timeframes = jgtcore.get_setting('_timeframes', 'D1')
```

### Environment Setup

```python
import jgtcore

# One-call environment setup
config, settings = jgtcore.setup_environment(demo=True)

# Check demo mode
if jgtcore.is_demo_mode():
    print("Running in demo mode")
```

---

## Configuration Lookup Order

### config.json
1. Current directory: `./config.json`
2. User home: `~/.jgt/config.json`
3. System: `/etc/jgt/config.json`
4. Environment: `JGT_CONFIG`, `JGT_CONFIG_PATH`, `JGT_CONFIG_JSON_SECRET`

### settings.json
1. System: `/etc/jgt/settings.json`
2. User home: `~/.jgt/settings.json`
3. Current directory: `./.jgt/settings.json`
4. YAML variants: `jgt.yml`, `_config.yml`
5. Environment: `JGT_SETTINGS`, `JGT_SETTINGS_PROCESS`

---

## Type Definitions

```python
from typing import Dict, Any, Optional, Tuple

def get_config(demo: bool = False) -> Dict[str, Any]: ...
def get_config_value(key: str, default: Any = None) -> Any: ...
def get_settings() -> Dict[str, Any]: ...
def get_setting(key: str, default: Any = None) -> Any: ...
def setup_environment(demo: bool = False) -> Tuple[Dict, Dict]: ...
def is_demo_mode() -> bool: ...
def readconfig(
    demo: bool = False, 
    export_env: bool = False,
    config_path: Optional[str] = None
) -> Dict[str, Any]: ...
def load_settings(custom_path: Optional[str] = None) -> Dict[str, Any]: ...
```

---

## Creative Advancement Scenarios

### Scenario: First-Time Package Access

**Desired Outcome**: New package imports jgtcore and immediately accesses broker credentials

**Current Reality**: Developer creates new trading tool needing config access

**Natural Progression**:
1. Import jgtcore: `import jgtcore`
2. Get config: `config = jgtcore.get_config()`
3. Access credentials: `user_id = config['user_id']`
4. Automatic file discovery and caching handles complexity

**Resolution**: Developer never writes file-reading logic; config "just works"

### Scenario: Demo/Live Mode Switching

**Desired Outcome**: Application seamlessly switches between demo and live credentials

**Current Reality**: Application starts with `--demo` flag or `JGT_DEMO=true`

**Natural Progression**:
1. Check mode: `jgtcore.is_demo_mode()`
2. Load appropriate config: `config = jgtcore.get_config(demo=True)`
3. Demo credentials replace live if present in config

**Resolution**: Single boolean controls entire credential set

---

## Module Structure

```
jgtcore/
├── __init__.py          # Public API exports
├── core.py              # Main configuration logic
├── constants.py         # Shared constants
├── compatibility.py     # Cross-version compatibility
├── env/                 # Environment detection
├── fx/                  # FX-specific helpers
├── logging/             # Logging configuration
├── os/                  # OS-specific utilities
└── timeframe.py         # Timeframe parsing
```

---

## Integration with JGT Ecosystem

```
jgtcore (this package)
    ↓ used by
jgtutils (adds CLI, utilities)
    ↓ used by
jgtapy (adds indicators)
    ↓ used by
jgtfxcon (adds broker connection)
    ↓ used by
jgtpy (adds data services)
    ↓ used by
jgtml (adds ML/analysis)
    ↓ served by
jgt-data-server (REST API)
    ↓ consumed by
jgt-code (terminal agent)
```

---

## Quality Criteria

✅ **Zero CLI Dependencies**: Pure library, no argparse/click required  
✅ **Cached Settings**: Single load per process, thread-safe  
✅ **Environment Export**: Config values become environment variables  
✅ **Demo Mode Support**: Built-in credential switching  
✅ **Hierarchical Lookup**: System → User → Process precedence

===============================================
FILE: core.spec.md
PATH: jgtcore/rispecs/core.spec.md
URL:  https://jgtcore.jgwill.com/rispecs/core.spec.md
===============================================

# jgtcore Specification

> Core Configuration and Tracing Infrastructure

**Specification Version**: 1.0  
**Module**: `jgtcore/core.py`, `jgtcore/tracer.py`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Centralized configuration management and tracing infrastructure that all JGT packages depend on for settings and observability.

**Achievement Indicator**: Any package can:
- Load settings from `~/.jgt/config.json`
- Override with environment variables
- Trace operations for debugging
- Access consistent timeframe utilities

**Value Proposition**: Foundation layer that ensures consistent behavior across all packages.

---

## Configuration Management

### Settings Loading

```python
def get_settings(
    args: Optional[argparse.Namespace] = None,
    force_refresh: bool = False
) -> Dict[str, Any]:
    """
    Load settings from multiple sources with priority.
    
    Priority Order (highest first):
        1. Command line arguments (args)
        2. Environment variables (JGT_*)
        3. User config (~/.jgt/config.json)
        4. Pattern config (~/.jgt/patterns/{pn}.json)
        5. Package defaults
    
    Cached after first load unless force_refresh=True.
    """

def _load_settings_from_path(path: str) -> Dict[str, Any]:
    """Load settings from JSON file."""

def _load_settings_from_path_yaml(
    path: str,
    key: Optional[str] = None
) -> Dict[str, Any]:
    """Load settings from YAML file (optional dependency)."""

def update_settings(
    old_settings: Dict[str, Any],
    new_settings: Dict[str, Any],
    keys: List[str] = ["patterns"]
) -> None:
    """Merge settings, handling special keys like 'patterns' separately."""
```

### Configuration Files

```
~/.jgt/
├── config.json           # Main configuration
├── config.yaml           # YAML alternative (optional)
├── patterns/
│   ├── mz.json           # Pattern: mz columns
│   └── full.json         # Pattern: full columns
└── secrets.json          # API tokens (not committed)
```

### Config Structure

```json
{
  "columns_to_remove": ["aocolor", "accolor"],
  "patterns": {
    "mz": {
      "columns": ["mfi_sig", "zone_sig", "ao"]
    }
  },
  "ttf_columns_to_remove": [],
  "default_quotescount": 335
}
```

---

## Tracing Infrastructure

### JGTTracer Class

```python
class JGTTracer:
    """
    Tracing infrastructure for observability.
    
    Usage:
        tracer = JGTTracer("jgtml", "fdb_scanner")
        with tracer.trace_operation("scan_all", metadata) as trace_id:
            tracer.add_step("step1", input_data={...})
            # ... operation ...
            tracer.add_step("step1_complete", output_data={...})
    """
    
    def __init__(
        self,
        package_name: str,
        component_name: str
    ):
        """Initialize tracer for a package/component."""
    
    def start_operation(
        self,
        operation_name: str,
        metadata: Dict[str, Any]
    ) -> str:
        """Start a traced operation, return trace_id."""
    
    def add_step(
        self,
        step_name: str,
        input_data: Optional[Dict] = None,
        output_data: Optional[Dict] = None
    ) -> None:
        """Add a step to the current trace."""
    
    def end_operation(
        self,
        trace_id: str,
        success: bool = True,
        error: Optional[str] = None
    ) -> None:
        """End a traced operation."""
    
    @contextmanager
    def trace_operation(
        self,
        operation_name: str,
        metadata: Dict[str, Any]
    ):
        """Context manager for traced operations."""
```

### Trace Output

```json
{
  "trace_id": "fdb_scan_260131_140500",
  "package": "jgtml",
  "component": "fdb_scanner",
  "operation": "scan_all",
  "start_time": "2026-01-31T14:05:00Z",
  "steps": [
    {
      "name": "load_instruments",
      "timestamp": "2026-01-31T14:05:01Z",
      "input": {"count": 28}
    },
    {
      "name": "scan_complete",
      "timestamp": "2026-01-31T14:06:30Z",
      "output": {"signals_found": 5}
    }
  ],
  "end_time": "2026-01-31T14:06:30Z",
  "success": true
}
```

---

## Timeframe Utilities

### Point of View (POV) Functions

```python
def get_higher_tf_array(timeframe: str) -> List[str]:
    """
    Get all timeframes higher than specified.
    
    Examples:
        "H1" -> ["H4", "D1", "W1", "MN"]
        "m15" -> ["m30", "H1", "H4", "D1", "W1", "MN"]
    """

def get_lower_tf_array(timeframe: str) -> List[str]:
    """Get all timeframes lower than specified."""

def is_higher_tf(tf1: str, tf2: str) -> bool:
    """Check if tf1 is higher than tf2."""

def get_tf_minutes(timeframe: str) -> int:
    """
    Get minutes for a timeframe.
    
    Examples:
        "m1" -> 1
        "H1" -> 60
        "D1" -> 1440
    """
```

### Timeframe Constants

```python
TIMEFRAMES = [
    "m1", "m5", "m15", "m30",
    "H1", "H2", "H3", "H4", "H6", "H8",
    "D1", "W1", "MN"
]

TF_MINUTES = {
    "m1": 1, "m5": 5, "m15": 15, "m30": 30,
    "H1": 60, "H2": 120, "H3": 180, "H4": 240,
    "H6": 360, "H8": 480, "D1": 1440,
    "W1": 10080, "MN": 43200
}
```

---

## Exception Handling

```python
def print_exception(e: Exception) -> None:
    """
    Print exception with traceback.
    
    Handles:
        - Standard exceptions
        - System exits
        - Keyboard interrupts
    """

def handle_keyboard_interrupt():
    """Clean handler for Ctrl+C."""
```

---

## Environment Variables

```python
# Data paths
JGTPY_DATA = os.getenv("JGTPY_DATA", "/src/jgtml/data/current")
JGTPY_DATA_FULL = os.getenv("JGTPY_DATA_FULL", "/var/lib/jgt/full/data")

# Configuration
JGT_CONFIG = os.getenv("JGT_CONFIG", "~/.jgt/config.json")

# Broker connection
FXCON_CONFIG = os.getenv("FXCON_CONFIG")

# Tracing
JGT_TRACE_ENABLED = os.getenv("JGT_TRACE_ENABLED", "false")
JGT_TRACE_OUTPUT = os.getenv("JGT_TRACE_OUTPUT", "~/.jgt/traces/")
```

---

## CLI Argument Parsing

```python
def new_parser(
    description: str,
    epilog: str = "",
    prog_name: str = "",
    add_exiting_quietly_flag: bool = False
) -> argparse.ArgumentParser:
    """Create new argument parser with JGT defaults."""

def add_instrument_timeframe_arguments(
    parser: argparse.ArgumentParser
) -> argparse.ArgumentParser:
    """Add -i and -t arguments."""

def add_use_fresh_argument(
    parser: argparse.ArgumentParser
) -> argparse.ArgumentParser:
    """Add --fresh argument."""

def add_verbose_argument(
    parser: argparse.ArgumentParser
) -> argparse.ArgumentParser:
    """Add -v/--verbose argument."""

def parse_args(
    parser: argparse.ArgumentParser
) -> argparse.Namespace:
    """Parse arguments with JGT defaults applied."""
```

---

## Dependencies

```python
import json
import os
import sys
import traceback
from datetime import datetime, time, timedelta, timezone
from typing import Any, Dict, Optional, Tuple

# Optional
import ruamel.yaml  # YAML support
```

---

## Quality Criteria

✅ **Central Config**: Single settings source for all packages  
✅ **Priority Cascade**: CLI > ENV > File > Defaults  
✅ **Tracing**: Full observability infrastructure  
✅ **TF Utilities**: Consistent timeframe handling  
✅ **Graceful Fallbacks**: Works without optional dependencies

===============================================
FILE: app.specs.md
PATH: jgt-data-server/rispecs/app.specs.md
===============================================

# JGT Data Server Application Specification

> Master specification for the Williams-Native Trading Intelligence Substrate

**Specification Version**: 1.0  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: A containerized data service that serves Williams 5 Dimensions as first-class data primitives, providing universe-aware perspectives for AI trading agents through REST API and MCP tools.

**Achievement Indicator**: Trading agents can query any instrument and receive multi-perspective Williams analysis (Signal Detection, Wave Analysis, Coordination) with automatic timeframe-based refresh.

**Value Proposition**: Bridge between jgtpy data processing and jgt-code terminal agent, serving processed trading intelligence rather than raw data.

---

## Application Overview

JGT Data Server is a Docker-containerized service that:
1. Serves Williams 5 Dimensions via REST API
2. Provides universe-aware perspectives for three-universe processing
3. Implements timeframe-based automatic refresh
4. Exposes MCP tools for AI agent integration
5. Wraps jgtpy data pipeline in production-ready API

---

## Structural Tension

**Current Reality**: jgtpy provides CLI-based data access; jgt-code needs API-based data access with universe-aware perspectives.

**Desired State**: REST API and MCP tools serve Williams intelligence in formats optimized for AI agent consumption.

**Natural Progression**: jgt-data-server transforms batch CLI operations into real-time API services aligned with jgt-code's three-universe architecture.

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│  Docker Compose Stack: jgt-data-server                              │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌─────────────────────────────────┐  ┌───────────────────────────┐ │
│  │  jgt-data-server (Python)       │  │  jgt-data-mcp (Node.js)   │ │
│  │  - FastAPI on :8080             │  │  - MCP Server on :8090    │ │
│  │  - Scheduler (timeframe-based)  │◄─│  - Wraps API for agents   │ │
│  │  - Williams 5 Dimensions        │  │  - jgt-code compatible    │ │
│  └─────────────────────────────────┘  └───────────────────────────┘ │
│                ▼                                                    │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │  Mounted Volumes                                                 ││
│  │  /data/current → pds/, cds/, ttf/, mlf/                         ││
│  │  /config → config.json (broker credentials)                     ││
│  └─────────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────────┘
```

---

## REST API Endpoints

### Health & Status

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/health` | GET | Health check |
| `/api/v1/status` | GET | Service status |
| `/api/v1/instruments` | GET | Configured instruments |
| `/api/v1/timeframes` | GET | Configured timeframes |

### Market Data

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/pds/{instrument}/{timeframe}` | GET | Price data |
| `/api/v1/cds/{instrument}/{timeframe}` | GET | Signal data |

### Williams 5 Dimensions

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/williams/dimensions/{instrument}/{timeframe}` | GET | All 5 dimensions |
| `/api/v1/williams/alligator/{instrument}/{timeframe}` | GET | Balance Line only |
| `/api/v1/williams/ao/{instrument}/{timeframe}` | GET | Awesome Oscillator |
| `/api/v1/williams/zones/{instrument}/{timeframe}` | GET | Zone analysis |

### Universe-Aware Perspectives

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/perspective/{instrument}` | GET | Full perspective |
| `/api/v1/perspective/{instrument}?universe=signal_detection` | GET | Signal focus |
| `/api/v1/perspective/{instrument}?universe=wave_analysis` | GET | Wave focus |
| `/api/v1/perspective/{instrument}?universe=coordination` | GET | Risk focus |

### Scheduler Control

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/refresh` | POST | Trigger manual refresh |
| `/api/v1/scheduler/status` | GET | Scheduler state |

### Chart Endpoints (Caishen-Compatible)

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/chart/cdb/{instrument}/{timeframe}` | GET | Full ChaosDataBuilder-style response |
| `/api/v1/chart/bars/{instrument}/{timeframe}` | GET | BarChaosItem array for charting |
| `/api/v1/chart/wheel/{instrument}` | GET | Medicine Wheel state |
| `/api/v1/chart/snapshot` | POST | Create THS snapshot |

These endpoints support Caishen POV format (EUR-USD_H4 with dash separator).

---

## MCP Tools

Tools available for AI agents via MCP protocol:

| Tool | Description |
|------|-------------|
| `get_market_data` | Get PDS/CDS/TTF/MLF data |
| `get_williams_dimensions` | Get all 5 Williams dimensions |
| `get_perspective` | Universe-aware perspective |
| `get_universe_view` | Specific universe data |
| `trigger_refresh` | Manual data refresh |
| `get_scheduler_status` | Scheduler state |
| `get_alligator_alignment` | HTF Alligator alignment |

### MCP Tool Examples

```typescript
// Get universe-specific view
const signalView = await mcpClient.call('get_universe_view', {
  instrument: 'EUR/USD',
  universe: 'signal_detection'
});

// Get full perspective
const perspective = await mcpClient.call('get_perspective', {
  instrument: 'EUR/USD',
  universe: 'all'
});
```

---

## Scheduler Logic

Automatic refresh based on timeframe completion:

| Timeframe | Refresh At |
|-----------|------------|
| m5 | :00, :05, :10, :15, :20, :25, :30, :35, :40, :45, :50, :55 |
| m15 | :00, :15, :30, :45 |
| m30 | :00, :30 |
| H1 | :00 (each hour) |
| H4 | 0:00, 4:00, 8:00, 12:00, 16:00, 20:00 |
| D1 | 00:00 UTC |

---

## Type Definitions

```python
from typing import Dict, Any, Optional, List
from dataclasses import dataclass
from enum import Enum

class Universe(Enum):
    SIGNAL_DETECTION = "signal_detection"
    WAVE_ANALYSIS = "wave_analysis"
    COORDINATION = "coordination"

@dataclass
class WilliamsDimensions:
    """Complete Williams 5 Dimensions snapshot"""
    instrument: str
    timeframe: str
    timestamp: str
    
    # Dimension 1: Fractal
    fractal_high: Optional[float]
    fractal_low: Optional[float]
    fdb_signal: Optional[str]  # "buy", "sell", None
    
    # Dimension 2: Momentum
    ao: float
    ao_color: str  # "green", "red"
    
    # Dimension 3: Acceleration
    ac: float
    ac_color: str
    
    # Dimension 4: Zone
    zone: str  # "green", "red", "gray"
    zone_signal: Optional[str]
    
    # Dimension 5: Balance Line (Alligator)
    jaw: float
    teeth: float
    lips: float
    alligator_state: str  # "sleeping", "awakening", "feeding", "sated"

@dataclass  
class UniversePerspective:
    """Universe-specific view of trading data"""
    universe: Universe
    instrument: str
    primary_signal: Optional[str]
    confidence: float
    details: Dict[str, Any]

@dataclass
class FullPerspective:
    """Combined three-universe perspective"""
    instrument: str
    signal_detection: UniversePerspective
    wave_analysis: UniversePerspective
    coordination: UniversePerspective
    lead_universe: Universe
    coherence_score: float
    recommendation: str
```

---

## Creative Advancement Scenarios

### Scenario: Agent Requests Trading Perspective

**Desired Outcome**: jgt-code agent receives three-universe analysis for EUR/USD

**Current Reality**: Agent needs to make trading decision

**Natural Progression**:
1. Agent calls: `get_perspective({ instrument: 'EUR/USD' })`
2. Server fetches latest CDS data
3. Builds signal_detection perspective (FDB status, zone)
4. Builds wave_analysis perspective (HTF alignment)
5. Builds coordination perspective (risk assessment)
6. Calculates coherence between universes
7. Returns combined perspective with recommendation

**Resolution**: Agent receives structured data for three-universe processing

### Scenario: Timeframe Bar Completion

**Desired Outcome**: Data automatically refreshed when H1 bar completes

**Current Reality**: 13:59:55 → 14:00:00 transition

**Natural Progression**:
1. Scheduler detects H1 boundary (14:00)
2. Triggers refresh for all H1-configured instruments
3. jgtpy cdscli refreshes CDS data
4. Cache updated with new signals
5. Webhook notifies interested agents (optional)

**Resolution**: Fresh data available within seconds of bar close

### Scenario: Docker Deployment

**Desired Outcome**: Production-ready data server running in container

**Current Reality**: Need to deploy jgt-data-server

**Natural Progression**:
1. Create `.env` from `.env.example`
2. Configure broker credentials in `~/.jgt/config.json`
3. Run: `docker compose up -d`
4. Verify: `curl http://localhost:8080/api/v1/health`
5. Connect jgt-code to MCP endpoint

**Resolution**: Server running with automatic refresh and MCP tools available

---

## Configuration

### Environment Variables

```bash
# Broker
JGT_CONFIG_DIR=~/.jgt
JGT_CONFIG_JSON_SECRET='{"user_id":...}'

# Data
JGTPY_DATA=./data/current

# Trading
JGTPY_SERVICE_INSTRUMENTS=EUR/USD,XAU/USD,SPX500
JGTPY_SERVICE_TIMEFRAMES=m5,m15,H1,H4,D1

# Service
JGT_DATA_PORT=8080
JGT_MCP_PORT=8090
JGTPY_SERVICE_DAEMON_MODE=true
```

### Docker Compose

```yaml
services:
  jgt-data-server:
    build: ./server
    ports:
      - "8080:8080"
    volumes:
      - ${JGTPY_DATA}:/data/current
      - ${JGT_CONFIG_DIR}:/config
    environment:
      - JGTPY_SERVICE_INSTRUMENTS=${INSTRUMENTS}
      - JGTPY_SERVICE_TIMEFRAMES=${TIMEFRAMES}
      
  jgt-data-mcp:
    build: ./mcp
    ports:
      - "8090:8090"
    depends_on:
      - jgt-data-server
```

---

## Module Structure

```
jgt-data-server/
├── docker-compose.yml    # Container orchestration
├── .env.example          # Environment template
├── server/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── pyproject.toml
│   └── src/
│       ├── main.py       # FastAPI application
│       ├── routes/       # API endpoints
│       ├── services/     # Business logic
│       └── scheduler/    # Timeframe scheduler
├── mcp/
│   ├── Dockerfile
│   └── src/              # MCP server (Node.js)
├── config/               # Configuration templates
└── data/                 # Mounted data directory
```

---

## Integration with JGT Ecosystem

```
jgtpy (data processing)
    ↓ provides CDS/TTF/MLF data
jgt-data-server (this package) ← API layer
    ↓ serves via REST/MCP
jgt-code (terminal agent)
    ↓ consumes for three-universe processing
Trading Decisions
```

---

## Quality Criteria

✅ **Williams Native**: 5 Dimensions as first-class API  
✅ **Universe Aware**: Three-universe perspectives built-in  
✅ **Auto Refresh**: Timeframe-based scheduling  
✅ **MCP Compatible**: Agent-friendly tool interface  
✅ **Docker Ready**: Production container deployment  
✅ **REST Complete**: Full API documentation at /docs

===============================================
FILE: fxaddorder.spec.md
PATH: jgtfxcon/rispecs/fxaddorder.spec.md
URL:  https://jgtfxcon.jgwill.com/rispecs/fxaddorder.spec.md
===============================================

# fxaddorder (Entry Order) Specification

> Create Entry Stop Orders on Broker

**Specification Version**: 1.0  
**Module**: `jgtfxcon/jgtfxentryorder.py`  
**CLI Command**: `fxaddorder`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Entry stop orders on the broker with specified entry rate, stop loss, direction, and position size.

**Achievement Indicator**: Running `fxaddorder --demo -i EUR/USD -bs B -rate 1.0950 -stop 1.0900 -lots 1` produces:
- Entry order created on broker
- Order ID returned
- Confirmation with rate and time-in-force

**Value Proposition**: Execute FDB signals by creating entry orders with pre-defined stop loss - the final step in the trading workflow.

---

## Structural Tension

**Current Reality**: FDB signal validated, entry/stop rates calculated, but no broker order exists.

**Desired State**: Entry order placed on broker, waiting for price to trigger.

**Natural Progression**: Parse args → Connect → Create request → Send order → Monitor → Confirm.

---

## CLI Interface

```python
def main():
    """
    JGT FX Create Entry Stop Order CLI.
    
    Arguments:
        --demo: Use demo account (vs real)
        -i, --instrument: Instrument symbol (required)
        -bs: Buy/Sell direction - "B" or "S" (required)
        -rate: Entry rate (required)
        -stop: Stop loss rate (required)
        -lots: Position size in lots (required)
    
    Examples:
        # Create buy entry order (demo)
        fxaddorder --demo -i EUR/USD -bs B -rate 1.0950 -stop 1.0900 -lots 1
        
        # Create sell entry order (demo)
        fxaddorder --demo -i EUR/USD -bs S -rate 1.0850 -stop 1.0900 -lots 2
        
        # Real account order
        fxaddorder -i GBP/USD -bs B -rate 1.2500 -stop 1.2450 -lots 0.5
    """
```

---

## Order Creation Flow

```python
def main():
    """
    Main entry point.
    
    Flow:
        1. Parse arguments
        2. Validate stop level provided
        3. Read broker credentials from config
        4. Connect to ForexConnect
        5. Get account (demo or real)
        6. Get offer for instrument
        7. Calculate amount from lots × base_unit_size
        8. Create ENTRY order request with:
           - OFFER_ID
           - ACCOUNT_ID
           - BUY_SELL
           - RATE (entry)
           - RATE_STOP (stop loss)
           - AMOUNT
        9. Set up OrdersMonitor listener
        10. Send request
        11. Wait for order confirmation
        12. Print order details
        13. Logout
    """
```

---

## OrdersMonitor Class

```python
class OrdersMonitor:
    """Monitor for tracking order creation."""
    
    def __init__(self):
        self.__order_id = None
        self.__orders = {}
        self.__event = Event()
    
    def on_added_order(self, _, __, order_row):
        """
        Callback when order is added.
        
        Stores order in internal dict and signals
        if this is the order we're waiting for.
        """
    
    def wait(self, time: float, order_id: str) -> Optional[OrderRow]:
        """
        Wait for order to appear with timeout.
        
        Returns order_row if found, None if timeout.
        """
    
    def find_order(self, order_id: str) -> Optional[OrderRow]:
        """Find order by ID in tracked orders."""
    
    def reset(self):
        """Clear tracked orders and reset event."""
```

---

## Order Parameters

```python
# Entry order type
entry = fxcorepy.Constants.Orders.ENTRY

# Order request parameters
request = fx.create_order_request(
    order_type=entry,
    OFFER_ID=offer.offer_id,      # Instrument offer
    ACCOUNT_ID=str_account,        # Trading account
    BUY_SELL=str_buy_sell,        # "B" or "S"
    RATE=str_rate,                # Entry price
    RATE_STOP=str_stop,           # Stop loss price
    AMOUNT=amount,                # Position size (units)
)
```

---

## Lot Size Calculation

```python
# Get base unit size for instrument
base_unit_size = trading_settings_provider.get_base_unit_size(
    str_instrument, account
)

# Calculate amount in units
amount = base_unit_size * str_lots

# Example:
# EUR/USD base_unit_size = 10000
# lots = 1
# amount = 10000 (0.1 standard lot)
```

---

## Error Handling

```python
# RequestFailedError handling
try:
    resp = fx.send_request(request)
except Exception as e:
    _type_of_exception = type(e).__name__
    
    if _type_of_exception == "RequestFailedError":
        print("Request failed Error Handling...")
        print("Input Stop value:", str_stop)
        # Often indicates invalid stop level (too close, wrong side)
        exit(1)
```

---

## Output

```
Adding a stop to an entry order with 
entry rate: 1.09500, stop: 1.09000
AccountID='12345678'
The order has been added. OrderID=456789, Type=SE, BuySell=B, Rate=1.09500, TimeInForce=GTC
```

---

## Dependencies

```python
from forexconnect import fxcorepy, ForexConnect, Common
from jgtutils import jgtconstants, jgtos, jgtcommon, jgtpov
import common_samples
from time import sleep
from threading import Event
```

---

## Quality Criteria

✅ **Entry + Stop**: Combined order with stop loss  
✅ **Lot Size Handling**: Proper unit conversion  
✅ **Demo/Real**: Account type selection  
✅ **Async Monitoring**: Wait for order confirmation  
✅ **Error Handling**: Clear messages for failures  
✅ **Timeout Protection**: 30-second wait limit

===============================================
FILE: fxreport.spec.md
PATH: jgtfxcon/rispecs/fxreport.spec.md
URL:  https://jgtfxcon.jgwill.com/rispecs/fxreport.spec.md
===============================================

# fxreport Specification

> Trading Account Reports from Broker

**Specification Version**: 1.0  
**Module**: `jgtfxcon/jgtfxreport.py`  
**CLI Command**: `fxreport`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: HTML trading reports downloaded from the broker for specified date range.

**Achievement Indicator**: Running `fxreport --demo` produces:
- Account balance displayed
- Report URL obtained
- HTML report saved to current directory

**Value Proposition**: Access broker trading reports for performance analysis and record keeping.

---

## Structural Tension

**Current Reality**: Trading history exists on broker but not accessible locally.

**Desired State**: HTML report saved locally for review and archiving.

**Natural Progression**: Parse args → Connect → Get accounts → Request report URL → Download → Save.

---

## CLI Interface

```python
def main():
    """
    JGT FX Report CLI.
    
    Arguments:
        --demo: Use demo account (vs real)
        --datefrom: Report start date (default: 1 month ago)
        --dateto: Report end date (default: today)
    
    Examples:
        # Default report (last month, demo)
        fxreport --demo
        
        # Specific date range
        fxreport --demo --datefrom 2026-01-01 --dateto 2026-01-31
        
        # Real account report
        fxreport --datefrom 2026-01-01
    """
```

---

## Report Generation Flow

```python
def main():
    """
    Main entry point.
    
    Flow:
        1. Parse arguments
        2. Read broker credentials from config
        3. Set date_from (default: 1 month ago)
        4. Set date_to (default: today)
        5. Connect to ForexConnect
        6. For each account:
           a. Get report URL from session
           b. Print account ID and balance
           c. Download HTML from URL
           d. Fix relative URLs to absolute
           e. Save to {account_id}.html
        7. Logout
    """
```

---

## Core Functions

```python
def get_reports(
    fc: ForexConnect,
    dt_from: datetime,
    dt_to: datetime
) -> None:
    """
    Download reports for all accounts.
    
    For each account:
        1. Get report URL via session.get_report_url()
        2. Print account info
        3. Download and save HTML
    """

def month_delta(
    date: datetime,
    delta: int
) -> datetime:
    """
    Calculate date offset by months.
    
    Handles month/year boundaries correctly.
    """
```

---

## Report Download

```python
# Get report URL from broker
url = fc.session.get_report_url(
    account.account_id,
    dt_from,
    dt_to,
    "html",       # Format
    None          # Additional params
)

# Download and process
response = urlopen(url)
report = response.read().decode('utf-8')

# Fix relative URLs to absolute
abs_path = '{0.scheme}://{0.netloc}/'.format(urlsplit(url))
report = re.sub(
    r'((?:src|href)=")[/\\](.*?")',
    r'\1' + abs_path + r'\2',
    report
)

# Save to file
with open(file_name, 'w') as file:
    file.write(report)
```

---

## Output

```
Obtaining report URL...
account_id=12345678; Balance=10532.25000
Report URL=https://fxcm.com/reports/...

Connecting...
OK
Downloading report...
Report is saved to 12345678.html
```

---

## Output File

- **Location**: Current working directory
- **Filename**: `{account_id}.html`
- **Content**: Complete HTML report with fixed resource URLs

---

## Dependencies

```python
from forexconnect import ForexConnect
from jgtutils import jgtconstants, jgtos, jgtcommon, jgtpov
import common_samples
import re
from urllib.parse import urlsplit
from urllib.request import urlopen
```

---

## Quality Criteria

✅ **Multi-Account**: Reports for all accounts  
✅ **Date Range**: Configurable start/end dates  
✅ **Default Range**: Sensible month default  
✅ **URL Fixing**: Relative paths made absolute  
✅ **Demo/Real**: Account type selection

===============================================
FILE: fxrmorder.spec.md
PATH: jgtfxcon/rispecs/fxrmorder.spec.md
URL:  https://jgtfxcon.jgwill.com/rispecs/fxrmorder.spec.md
===============================================

# fxrmorder (Remove Order) Specification

> Delete Pending Orders from Broker

**Specification Version**: 1.0  
**Module**: `jgtfxcon/jgtfxremoveorder.py`  
**CLI Command**: `fxrmorder`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Removal of pending entry orders from the broker by order ID.

**Achievement Indicator**: Running `fxrmorder --demo -id 456789` produces:
- Order deleted from broker
- Confirmation message with order ID

**Value Proposition**: Cancel pending orders when signal is invalidated or trading plan changes.

---

## Structural Tension

**Current Reality**: Pending order exists on broker but is no longer wanted.

**Desired State**: Order removed, capital freed for other opportunities.

**Natural Progression**: Parse args → Connect → Find order → Delete request → Confirm deletion.

---

## CLI Interface

```python
def main():
    """
    JGT FX Remove Entry Order CLI.
    
    Arguments:
        --demo: Use demo account (vs real)
        -id, --orderid: Order ID to delete (required)
    
    Examples:
        # Remove order (demo)
        fxrmorder --demo -id 456789
        
        # Remove order (real)
        fxrmorder -id 456789
    """
```

---

## Order Deletion Flow

```python
def main():
    """
    Main entry point.
    
    Flow:
        1. Parse arguments
        2. Read broker credentials from config
        3. Connect to ForexConnect
        4. Query orders table by order_id
        5. If order not found: raise exception
        6. Create DELETE_ORDER request with:
           - COMMAND: DELETE_ORDER
           - ACCOUNT_ID: from order
           - ORDER_ID: from args
        7. Set up OrdersMonitor listener (on_delete)
        8. Send request
        9. Wait for deletion confirmation
        10. Print confirmation
        11. Logout
    """
```

---

## OrdersMonitor Class

```python
class OrdersMonitor:
    """Monitor for tracking order deletion."""
    
    def __init__(self):
        self.__order_id = None
        self.__deleted_orders = {}
        self.__event = threading.Event()
    
    def on_delete_order(self, _, __, order_row):
        """
        Callback when order is deleted.
        
        Stores deleted order and signals if this
        is the order we're waiting for.
        """
    
    def wait(self, time: float, order_id: str) -> Optional[OrderRow]:
        """
        Wait for order deletion with timeout.
        
        Returns order_row if deleted, None if timeout.
        """
    
    def find_order(self, order_id: str) -> Optional[OrderRow]:
        """Find order by ID in deleted orders."""
    
    def reset(self):
        """Clear tracked orders and reset event."""
```

---

## Delete Request

```python
# Find the order first
orders_table = fx.get_table(ForexConnect.ORDERS)
orders = orders_table.get_rows_by_column_value("order_id", order_id)

for order_row in orders:
    order = order_row
    break

if order is None:
    raise Exception(f"Order {order_id} not found")

# Create delete request
request = fx.create_request({
    fxcorepy.O2GRequestParamsEnum.COMMAND: fxcorepy.Constants.Commands.DELETE_ORDER,
    fxcorepy.O2GRequestParamsEnum.ACCOUNT_ID: order.account_id,
    fxcorepy.O2GRequestParamsEnum.ORDER_ID: str_old
})
```

---

## Output

```
The order has been deleted. Order ID: 456789
```

---

## Error Cases

| Scenario | Message |
|----------|---------|
| Order not found | "Order {id} not found" |
| Already filled | Cannot delete (order converted to trade) |
| Timeout | "Response waiting timeout expired." |

---

## Dependencies

```python
from forexconnect import fxcorepy, ForexConnect, Common
from jgtutils import jgtconstants, jgtos, jgtcommon, jgtpov
import common_samples
import threading
from time import sleep
```

---

## Quality Criteria

✅ **Order Lookup**: Verify order exists before delete  
✅ **Async Monitoring**: Wait for deletion confirmation  
✅ **Timeout Protection**: 30-second wait limit  
✅ **Demo/Real**: Account type selection  
✅ **Clear Output**: Confirmation message

===============================================
FILE: fxtransact.spec.md
PATH: jgtfxcon/rispecs/fxtransact.spec.md
URL:  https://jgtfxcon.jgwill.com/rispecs/fxtransact.spec.md
===============================================

# fxtr (Transaction Manager) Specification

> Trading Positions and Orders Management

**Specification Version**: 1.0  
**Module**: `jgtfxcon/jgtfxtransact.py`  
**CLI Command**: `fxtr`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Complete visibility into active trades and pending orders from the broker, with filtering and JSON export capabilities.

**Achievement Indicator**: Running `fxtr --demo -table all` produces:
- Account balance display
- List of all open trades with details
- List of all pending orders with details
- JSON export of transaction data

**Value Proposition**: Single command to see trading position state, essential for order management and risk assessment.

---

## Structural Tension

**Current Reality**: Trades and orders exist on broker but not visible in trading system.

**Desired State**: Complete transaction snapshot accessible via CLI with structured output.

**Natural Progression**: Connect → Query tables → Parse rows → Structure data → Output/Save.

---

## CLI Interface

```python
def main():
    """
    JGT FX Transact CLI.
    
    Arguments:
        --demo: Use demo account (vs real)
        -table: Table to display
            - "orders": Pending orders only
            - "trades": Open trades only
            - "all": Both tables (default)
        -account: Filter by account ID
        -id, --orderid: Filter by order/trade ID
        -i, --instrument: Filter by instrument
        -save: Save output to JSON file
    
    Examples:
        # Show all trades and orders (demo)
        fxtr --demo -table all
        
        # Show only pending orders
        fxtr --demo -table orders
        
        # Filter by instrument
        fxtr --demo -i EUR/USD
        
        # Save to file
        fxtr --demo -table trades -save
    """
```

---

## Data Structures

### FXOrder

```python
@dataclass
class FXOrder:
    """Represents a pending order."""
    order_id: str
    account_id: str
    offer_id: str
    instrument: str
    buy_sell: str              # "B" or "S"
    rate: float                # Entry rate
    rate_stop: float           # Stop rate (if set)
    rate_limit: float          # Limit rate (if set)
    amount: int                # Position size
    time_in_force: str         # Order duration
    status: str                # Order status
    
    @classmethod
    def from_string(cls, string: str) -> 'FXOrder':
        """Parse from ForexConnect row string."""
    
    def tojson(self) -> str:
        """Serialize to JSON."""
```

### FXTrade

```python
@dataclass
class FXTrade:
    """Represents an open trade."""
    trade_id: str
    account_id: str
    offer_id: str
    instrument: str
    buy_sell: str              # "B" or "S"
    open_rate: float           # Entry price
    amount: int                # Position size
    pl: float                  # Profit/Loss
    gross_pl: float            # Gross P/L
    stop: float                # Stop loss rate
    limit: float               # Take profit rate
    open_time: datetime        # Open timestamp
    
    @classmethod
    def from_string(cls, string: str) -> 'FXTrade':
        """Parse from ForexConnect row string."""
    
    def tojson(self) -> str:
        """Serialize to JSON."""
```

### FXTransactWrapper

```python
class FXTransactWrapper:
    """Container for orders and trades."""
    orders: FXOrders
    trades: FXTrades
    
    def add_orders(self, orders: FXOrders) -> None:
        """Add orders collection."""
    
    def add_trades(self, trades: FXTrades) -> None:
        """Add trades collection."""
    
    def tojson(self) -> str:
        """Serialize all to JSON."""
    
    def tojsonfile(self, filename: str) -> None:
        """Save to JSON file."""
```

---

## Core Flow

```python
def main():
    """
    Main entry point.
    
    Flow:
        1. Parse arguments
        2. Read broker credentials from config
        3. Connect to ForexConnect
        4. Get account information
        5. If table == "orders" or "all":
           - Query orders table
           - Parse each order row
           - Filter by ID/instrument if specified
           - Add to FXOrders collection
        6. If table == "trades" or "all":
           - Query trades table
           - Parse each trade row
           - Filter by ID/instrument if specified
           - Add to FXTrades collection
        7. Build FXTransactWrapper
        8. If -save: write to JSON file
        9. Logout
    """
```

---

## Table Parsing

```python
def parse_orders(
    table_manager,
    account_id: str
) -> FXOrders:
    """
    Parse orders table from ForexConnect.
    
    Returns FXOrders collection with all matching orders.
    """

def parse_trades(
    table_manager,
    account_id: str
) -> FXTrades:
    """
    Parse trades table from ForexConnect.
    
    Returns FXTrades collection with all matching trades.
    """

def _order_row_to_string(order_row) -> str:
    """Convert ForexConnect order row to parseable string."""

def _trade_row_to_string(trade_row, trade_data: dict) -> str:
    """Convert ForexConnect trade row to parseable string."""
```

---

## File Output

```python
def save_fxtransact_to_file(
    fxtransactwrapper: FXTransactWrapper,
    str_table: str = "all",
    str_connection: str = "",
    save_prefix: str = "fxtransact_"
) -> None:
    """
    Save transaction data to JSON file.
    
    Filename patterns:
        - All: {connection}_fxtransact_.json
        - Orders: {connection}_fxtransact_orders.json
        - Trades: {connection}_fxtransact_trades.json
        - By ID: {connection}_fxtransact_{order_id}.json
        - By Instrument: {connection}_fxtransact_{instrument}.json
    """
```

---

## Output Format

```json
{
    "orders": [
        {
            "order_id": "123456",
            "account_id": "12345678",
            "instrument": "EUR/USD",
            "buy_sell": "B",
            "rate": 1.0950,
            "rate_stop": 1.0900,
            "amount": 10000,
            "time_in_force": "GTC",
            "status": "Waiting"
        }
    ],
    "trades": [
        {
            "trade_id": "789012",
            "account_id": "12345678",
            "instrument": "GBP/USD",
            "buy_sell": "S",
            "open_rate": 1.2500,
            "amount": 20000,
            "pl": 150.25,
            "stop": 1.2600,
            "limit": 1.2300
        }
    ]
}
```

---

## Dependencies

```python
from forexconnect import ForexConnect, EachRowListener
from jgtutils import jgtconstants, jgtos, jgtcommon, jgtpov
from jgtutils.FXTransact import (
    FXTransactWrapper, FXOrder, FXOrders,
    FXTrade, FXTrades
)
import common_samples
```

---

## Quality Criteria

✅ **Dual Tables**: Orders and trades in one command  
✅ **Filtering**: By ID, instrument, or account  
✅ **JSON Export**: Structured output for automation  
✅ **Demo/Real**: Switch between account types  
✅ **Account Display**: Balance and ID shown  
✅ **Complete Data**: All order/trade fields captured

===============================================
FILE: jgtfxcli.spec.md
PATH: jgtfxcon/rispecs/jgtfxcli.spec.md
URL:  https://jgtfxcon.jgwill.com/rispecs/jgtfxcli.spec.md
===============================================

# jgtfxcli Specification

> Price Data CLI for Broker Connectivity

**Specification Version**: 1.0  
**Module**: `jgtfxcon/jgtfxcli.py`  
**CLI Command**: `jgtfxcli`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Fresh OHLCV price data files downloaded from the broker and cached locally, ready for indicator calculations and signal detection.

**Achievement Indicator**: Running `jgtfxcli -i EUR/USD -t H1` produces:
- Price data fetched from broker API
- CSV saved to `$JGTPY_DATA/pds/EUR-USD_H1.csv`
- DataFrame with Date index, OHLCV columns, optional Bid/Ask

**Value Proposition**: Single command to refresh price data for any instrument/timeframe combination.

---

## Structural Tension

**Current Reality**: No local price data, or stale cached data.

**Desired State**: Fresh OHLCV data in standardized format, ready for analysis pipeline.

**Natural Progression**: Parse args → Connect broker → Fetch data → Transform → Cache → Return.

---

## CLI Interface

```python
def main():
    """
    JGT Price History CLI.
    
    Arguments:
        -i, --instrument: Instrument symbol(s), comma-separated
        -t, --timeframe: Timeframe(s), comma-separated
        -n, --quotescount: Number of bars to fetch
        --full: Use full historical data mode
        --tlidrange: TLID range for specific date range
        --viewpath: Show output path only (no fetch)
        --compress: Use gzip compression
        --keepbidask: Preserve Bid/Ask columns
        --dropna-volume: Drop zero-volume bars
        --server: Start PDS server (deprecated)
        --iprop: Download instrument properties (deprecated)
        -v, --verbose: Verbosity level (0-2)
        --debug: Enable debug mode
    
    Examples:
        # Single instrument/timeframe
        jgtfxcli -i EUR/USD -t H1
        
        # Multiple instruments and timeframes
        jgtfxcli -i EUR/USD,GBP/USD -t H1,H4,D1
        
        # Full historical data
        jgtfxcli -i SPX500 -t D1 --full
        
        # View output path only
        jgtfxcli -i EUR/USD -t H1 --viewpath
        
        # Keep bid/ask columns
        jgtfxcli -i EUR/USD -t H1 --keepbidask
    """
```

---

## Core Function Flow

```python
def main():
    """
    Main entry point.
    
    Flow:
        1. Parse arguments
        2. Split instruments and timeframes (comma-separated)
        3. For each instrument/timeframe pair:
           a. Call JGTPDSSvc.getPHs()
           b. Connect to broker if not connected
           c. Fetch price history
           d. Transform columns (Bid/Ask → OHLC)
           e. Write to cache file
        4. Print output paths
    """
```

---

## Service Layer: JGTPDSSvc

```python
def getPHs(
    instrument: str,              # Single or comma-separated
    timeframe: str,               # Single or comma-separated
    quote_count: int = -1,        # Bars to fetch
    start: datetime = None,       # Start date
    end: datetime = None,         # End date
    with_index: bool = True,      # Include date index
    quiet: bool = True,           # Suppress output
    compressed: bool = False,     # Gzip output
    tlid_range: str = None,       # TLID date range
    use_full: bool = False,       # Full historical mode
    default_quote_count: int = 335,
    default_add_quote_count: int = 89,
    verbose_level: int = 0,
    view_output_path_only: bool = False,
    keep_bid_ask: bool = False,
    dropna_volume: bool = True
) -> List[str]:
    """
    Fetch price history for multiple instruments/timeframes.
    
    Algorithm:
        1. Parse comma-separated instruments/timeframes
        2. Set stayConnected=True (single broker connection)
        3. For each pair: call getPH()
        4. Disconnect when complete
        5. Return list of updated POVs
    """

def getPH(
    instrument: str,
    timeframe: str,
    quote_count: int = -1,
    ...
) -> Tuple[str, pd.DataFrame]:
    """
    Fetch price history for single instrument/timeframe.
    
    Algorithm:
        1. If view_output_path_only: return path, None
        2. If use_full: calculate quote count for timeframe
        3. Convert TLID range to datetime if provided
        4. Call JGTPDS.getPH2file()
        5. If exception: try alternative command (fxcli2console)
        6. Return (filepath, dataframe)
    """
```

---

## Quote Count Calculation

```python
# For full historical mode, calculate proportional quote count
# Based on M1 as reference (pov_full_M1 default: 1000)

TIMEFRAME_RATIOS = {
    "m1": 1,
    "m5": 5,
    "m15": 15,
    "m30": 30,
    "H1": 60,
    "H4": 240,
    "D1": 1440,
    "W1": 10080,
    "MN": 43200
}

def calculate_quote_counts_tf(m1_count: int) -> Dict[str, int]:
    """
    Calculate quote counts for all timeframes based on M1.
    
    Example with m1_count=1000:
        m1: 1000, m5: 200, H1: 16, D1: 1, ...
    """
```

---

## Alternative Command Fallback

```python
def _run_get_ph_using_alt_command(
    instrument: str,
    timeframe: str,
    use_full: bool,
    tlid_range: str,
    compressed: bool,
    quiet: bool,
    fxcli2console: str = "fxcli2console",
    keep_bid_ask: bool = False
) -> Tuple[bool, str]:
    """
    Fallback to fxcli2console if primary fetch fails.
    
    Runs: fxcli2console -i {instrument} -t {timeframe} [-kba]
    Output redirected to: $JGTPY_DATA/pds/{instrument}_{timeframe}.csv
    
    Used when ForexConnect fails but external tool works.
    """
```

---

## Output Paths

```python
def _make_output_fullpath(
    instrument: str,
    timeframe: str,
    use_full: bool,
    tlid_range: str,
    compress: bool,
    quiet: bool
) -> str:
    """
    Generate output file path.
    
    Patterns:
        Current: $JGTPY_DATA/pds/{i}_{t}.csv
        Full: $JGTPY_DATA_FULL/pds/{i}_{t}.csv
        TLID: $JGTPY_DATA/pds/{i}_{t}_{tlid_from}_{tlid_to}.csv
        Compressed: {path}.gz
    
    Example:
        /src/jgtml/data/current/pds/EUR-USD_H1.csv
    """
```

---

## Configuration

```python
# Environment variables
JGTPY_DATA = "/src/jgtml/data/current"
JGTPY_DATA_FULL = "/var/lib/jgt/full/data"
pov_full_M1 = 1000  # Reference bars for M1
JGT_KEEP_BID_ASK = "0"  # Override keep_bid_ask flag
RUN_ALT = "1"  # Enable alternative command fallback

# Config file (~/.jgt/config.json)
{
    "keep_bid_ask": false,
    "columns_to_remove": [...]
}
```

---

## Output Schema

```python
# Standard PDS DataFrame
{
    "Date": datetime,      # Index
    "Open": float,         # (BidOpen + AskOpen) / 2
    "High": float,         # (BidHigh + AskHigh) / 2
    "Low": float,          # (BidLow + AskLow) / 2
    "Close": float,        # (BidClose + AskClose) / 2
    "Volume": int,         # Trade volume
    
    # Optional (--keepbidask)
    "BidOpen": float,
    "BidHigh": float,
    "BidLow": float,
    "BidClose": float,
    "AskOpen": float,
    "AskHigh": float,
    "AskLow": float,
    "AskClose": float,
}
```

---

## Dependencies

```python
import pandas as pd
from jgtutils import jgtconstants, jgtos, jgtcommon, jgtpov
import JGTPDS as pds
import JGTPDSSvc as svc
```

---

## Quality Criteria

✅ **Multi-POV Support**: Comma-separated instruments/timeframes  
✅ **Full Mode**: Proportional quote counts for historical data  
✅ **Fallback**: Alternative command if primary fails  
✅ **Bid/Ask Preservation**: Optional detailed prices  
✅ **View Path**: Check output without fetching  
✅ **Zero-Volume Drop**: Clean data quality

===============================================
FILE: pds.spec.md
PATH: jgtfxcon/rispecs/pds.spec.md
URL:  https://jgtfxcon.jgwill.com/rispecs/pds.spec.md
===============================================

# PDS (Price Data Service) Specification

> OHLCV Price Data Acquisition and Storage

**Specification Version**: 1.0  
**Modules**: `jgtfxcon/JGTPDS.py`, `jgtfxcon/jgtfxc.py`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Clean OHLCV price data for any instrument/timeframe, retrieved from broker API or cached files, with proper column structure ready for indicator calculations.

**Achievement Indicator**: Calling `pds.getPH("EUR/USD", "H1")` produces:
- DataFrame with Date index
- OHLCV columns (Open, High, Low, Close, Volume)
- Optional Bid/Ask columns preserved
- Data from cache or fresh from broker

**Value Proposition**: Single interface for price data regardless of source - abstracts broker connectivity and caching.

---

## Structural Tension

**Current Reality**: Raw broker API returns data in various formats, connectivity required.

**Desired State**: Consistent DataFrame format with OHLCV columns, available offline from cache.

**Natural Progression**: Connect → Fetch → Transform → Cache → Return.

---

## Data Pipeline Position

```
[PDS] → IDS → CDS → TTF → MLF → MX
  ↑
Source of all data
```

**Dependencies**: jgtfxc for broker connectivity, jgtutils for file operations.

---

## Core Function: getPH

```python
def getPH(
    instrument: str,
    timeframe: str,
    quotes_count: int = -1,
    date_from: datetime = None,
    date_to: datetime = None,
    compressed: bool = False,
    quiet: bool = True,
    keep_bid_ask: bool = True
) -> pd.DataFrame:
    """
    Get Price History for instrument/timeframe.
    
    Args:
        instrument: Symbol (e.g., "EUR/USD", "SPX500")
        timeframe: Period (e.g., "m1", "H1", "D1")
        quotes_count: Number of bars to fetch (-1 = default)
        date_from: Start date for range
        date_to: End date for range
        compressed: Read/write gzip format
        quiet: Suppress output
        keep_bid_ask: Preserve Bid/Ask columns
    
    Logic:
        1. Check if cached file exists
        2. If cache valid and not fresh requested: read cache
        3. Otherwise: connect to broker, fetch, cache, return
    
    Returns:
        DataFrame with Date index and OHLCV columns
    """
```

---

## Data Sources

### Broker API (Fresh Data)

```python
def connect(quiet: bool = True, json_config_str: str = None):
    """
    Establish broker connection.
    
    Config loaded from:
        ~/.jgt/config.json or FXCON_CONFIG environment
    
    Connection parameters:
        - access_token: API authentication
        - account_id: Trading account
        - server: "demo" or "real"
    """

def disconnect(quiet: bool = True):
    """Close broker connection."""

def status(quiet: bool = True) -> bool:
    """Check if connected."""
```

### Filestore (Cached Data)

```python
def getPH_from_filestore(
    instrument: str,
    timeframe: str,
    quiet: bool = True,
    compressed: bool = False,
    with_index: bool = True,
    tlid_range: str = None,
    output_path: str = None,
    keep_bid_ask: bool = False
) -> pd.DataFrame:
    """
    Read cached PDS from filestore.
    
    File Paths:
        $JGTPY_DATA/pds/{instrument}_{timeframe}.csv
        Or with TLID range: {instrument}_{timeframe}_{tlid_range}.csv
    """
```

---

## Column Transformation

### Standard OHLC from Bid/Ask

```python
def pds_add_ohlc_stc_columns(
    dfsrc: pd.DataFrame,
    rounding_nb: int = 10
) -> pd.DataFrame:
    """
    Create standard OHLC from Bid/Ask columns.
    
    Transformation:
        Open = (BidOpen + AskOpen) / 2
        High = (BidHigh + AskHigh) / 2
        Low = (BidLow + AskLow) / 2
        Close = (BidClose + AskClose) / 2
        Median = (High + Low) / 2
    
    Rounding applied to prevent float precision issues.
    """
```

### Column Cleansing

```python
def _cleanse_original_columns(
    dfsrc: pd.DataFrame,
    quiet: bool = True
) -> pd.DataFrame:
    """
    Remove broker-specific columns not needed for analysis.
    
    Removed columns typically include:
        - tickqty (tick quantity)
        - Status, State columns
        - Internal broker identifiers
    """
```

---

## Output Schema

```python
{
    # Index
    "Date": datetime,          # Timestamp index
    
    # Standard OHLCV
    "Open": float,             # Opening price
    "High": float,             # Highest price
    "Low": float,              # Lowest price
    "Close": float,            # Closing price
    "Volume": int,             # Trade volume
    
    # Derived
    "Median": float,           # (High + Low) / 2
    
    # Bid/Ask (if keep_bid_ask=True)
    "BidOpen": float,
    "BidHigh": float,
    "BidLow": float,
    "BidClose": float,
    "AskOpen": float,
    "AskHigh": float,
    "AskLow": float,
    "AskClose": float,
}
```

---

## File Operations

### Read from File

```python
def read_ohlc_df_from_file(
    srcpath: str,
    quiet: bool = True,
    compressed: bool = False,
    with_index: bool = True,
    keep_bid_ask: bool = False
) -> pd.DataFrame:
    """
    Read OHLC DataFrame from CSV file.
    
    Supports:
        - Plain CSV
        - Gzip compressed CSV
        - Date as index or column
        - Bid/Ask column retention
    """
```

### Write to File

```python
def write_pds_to_file(
    df: pd.DataFrame,
    instrument: str,
    timeframe: str,
    compressed: bool = False,
    output_path: str = None
) -> str:
    """
    Write PDS DataFrame to cache file.
    
    Returns:
        File path where data was written
    """
```

---

## Timeframe Mapping

```python
TIMEFRAMES = {
    "m1": "1 minute",
    "m5": "5 minutes",
    "m15": "15 minutes",
    "m30": "30 minutes",
    "H1": "1 hour",
    "H2": "2 hours",
    "H3": "3 hours",
    "H4": "4 hours",
    "H6": "6 hours",
    "H8": "8 hours",
    "D1": "1 day",
    "W1": "1 week",
    "MN": "1 month"
}
```

---

## Instrument Properties

```python
# From iprops module
def get_pips(instrument: str) -> float:
    """
    Get pip size for instrument.
    
    Examples:
        "EUR/USD" -> 0.0001
        "USD/JPY" -> 0.01
        "SPX500" -> 1.0
    """

def get_precision(instrument: str) -> int:
    """Get decimal precision for prices."""
```

---

## File Locations

```python
# Environment variables:
JGTPY_DATA = "/src/jgtml/data/current"      # Current data
JGTPY_DATA_FULL = "/src/jgtml/data/full"    # Full historical

# PDS file paths:
$JGTPY_DATA/pds/{instrument}_{timeframe}.csv

# With TLID range:
$JGTPY_DATA/pds/{instrument}_{timeframe}_{tlid_from}_{tlid_to}.csv
```

---

## Error Handling

```python
# Connection errors
class ConnectionError(Exception):
    """Raised when broker connection fails."""

# Data errors
class DataNotFoundError(Exception):
    """Raised when requested data not available."""

# File errors  
class FileNotFoundError(Exception):
    """Raised when cache file missing and offline."""
```

---

## Dependencies

```python
import pandas as pd
import datetime
from jgtfxcon import jgtfxc as jfx      # Broker connectivity
from jgtfxcon import JGTPDHelper as jpd  # DataFrame helpers
from jgtfxcon import iprops              # Instrument properties
from jgtutils import jgtos               # File operations
from jgtutils import jgtconstants as c   # Constants
```

---

## Quality Criteria

✅ **Dual Source**: Broker API and file cache  
✅ **Column Standardization**: Consistent OHLCV format  
✅ **Bid/Ask Preservation**: Optional detailed prices  
✅ **Compression Support**: Gzip for storage efficiency  
✅ **Date Indexing**: Proper datetime index  
✅ **Connection Pooling**: Stay connected for batch operations

===============================================
FILE: alligator-analysis.spec.md
PATH: jgtml/rispecs/alligator-analysis.spec.md
URL:  https://jgtml.jgwill.com/rispecs/alligator-analysis.spec.md
===============================================

# Alligator Analysis Specification

> Multi-Period Alligator Convergence Analysis

**Specification Version**: 1.0  
**Module**: `jgtml/TideAlligatorAnalysis.py`  
**CLI**: `jgtml/alligator_cli.py`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Comprehensive Alligator analysis across three period scales (Regular, Big, Tide), revealing market structure from quick direction changes to macro trends, with convergence scoring for high-probability setups.

**Achievement Indicator**: Running `python -m jgtml.alligator_cli -i SPX500 -t D1 --type all` produces:
- Regular Alligator (5-8-13) state analysis
- Big Alligator (34-55-89) intermediate cycle analysis
- Tide Alligator (144-233-377) macro trend analysis
- Convergence score showing alignment across all three

**Value Proposition**: See the complete market structure at once - from micro direction to macro trend - enabling trades aligned with multiple timeframes of momentum.

---

## Structural Tension

**Current Reality**: Trader sees only regular Alligator on chart, missing larger cycle context.

**Desired State**: Complete three-scale Alligator view showing where current price sits within the fractal structure of the market.

**Natural Progression**: Load CDS → Calculate all Alligator periods → Analyze mouth states → Score convergence → Report alignment.

---

## Three Alligator Scales

### Regular Alligator (5-8-13)

**Periods**: Lips=5, Teeth=8, Jaw=13  
**Shifts**: Lips=3, Teeth=5, Jaw=8  
**Columns**: `lips`, `teeth`, `jaw`

**Use Case**: Quick market direction detection, day trading entries

**Mouth States**:
- **Sleeping**: Lines intertwined, no trend
- **Awakening**: Lines separating, trend starting
- **Feeding**: Lines spread apart, strong trend
- **Sated**: Lines converging, trend ending

### Big Alligator (34-55-89)

**Periods**: Lips=34, Teeth=55, Jaw=89  
**Columns**: `blips`, `bteeth`, `bjaw`

**Use Case**: Intermediate cycle analysis, swing trading

**Analysis Focus**:
- Weekly/monthly cycle position
- Larger wave structure
- Support/resistance from Big Alligator lines

### Tide Alligator (144-233-377)

**Periods**: Lips=144, Teeth=233, Jaw=377  
**Columns**: `tlips`, `tteeth`, `tjaw`

**Use Case**: Macro trend identification, position trading

**Analysis Focus**:
- Monthly/quarterly trend direction
- Major support/resistance levels
- Macro wave position

---

## Type Definitions

```python
from enum import Enum
from dataclasses import dataclass
from typing import Optional, List, Dict

class AlligatorType(Enum):
    """Three Alligator analysis types"""
    REGULAR = "normal"    # 5-8-13 periods
    BIG = "big"           # 34-55-89 periods
    TIDE = "tide"         # 144-233-377 periods

class MouthState(Enum):
    """Alligator mouth states"""
    SLEEPING = "sleeping"      # Lines intertwined
    AWAKENING = "awakening"    # Lines separating
    FEEDING = "feeding"        # Lines spread apart
    SATED = "sated"           # Lines converging

class Direction(Enum):
    """Market direction"""
    BULLISH = "bullish"
    BEARISH = "bearish"
    NEUTRAL = "neutral"

@dataclass
class AlligatorState:
    """State of a single Alligator"""
    type: AlligatorType
    jaw: float
    teeth: float
    lips: float
    mouth_state: MouthState
    direction: Direction
    spread: float              # Distance between lines
    
@dataclass
class ConvergenceAnalysis:
    """Analysis of all three Alligators"""
    instrument: str
    timeframe: str
    regular: AlligatorState
    big: AlligatorState
    tide: AlligatorState
    convergence_score: float   # 0.0 to 1.0
    aligned_direction: Optional[Direction]
    recommendation: str
```

---

## Configuration

```python
@dataclass
class AlligatorConfig:
    """Configuration for Alligator Analysis"""
    instrument: str = 'SPX500'
    timeframe: str = 'D1'
    alligator_types: List[AlligatorType] = None  # Default: all three
    force_regenerate_mxfiles: bool = True
    mfi_flag: bool = True
    regenerate_cds: bool = True
    use_fresh: bool = True
    quiet: bool = False
    jgtdroot: str = "/b/Dropbox/jgt"
    drop_subdir: str = "drop"
    result_file_basename: str = "jgtml_alligator_analysis.result"
```

---

## Core Analysis Class

```python
class AlligatorAnalysis:
    """
    Unified Alligator Analysis supporting all three types.
    """
    
    def __init__(self, config: AlligatorConfig):
        self.config = config
        self.data_cache = {}
        self.results_cache = {}
    
    def analyze(self, direction: str = "B") -> ConvergenceAnalysis:
        """
        Run full analysis for specified direction.
        
        Args:
            direction: "B" for buy analysis, "S" for sell analysis
        
        Returns:
            ConvergenceAnalysis with all three Alligator states
        """
    
    def get_alligator_state(
        self, 
        alligator_type: AlligatorType,
        bar: pd.Series
    ) -> AlligatorState:
        """
        Get state of specific Alligator from bar data.
        
        Determines mouth state by analyzing:
        - Line order (lips/teeth/jaw)
        - Line spread (distance between lines)
        - Direction (bullish/bearish/neutral)
        """
    
    def calculate_convergence_score(
        self,
        regular: AlligatorState,
        big: AlligatorState,
        tide: AlligatorState
    ) -> float:
        """
        Calculate convergence score (0.0 to 1.0).
        
        Higher score = more alignment between all three Alligators
        
        Scoring:
        - Same direction for all three: +0.5
        - All mouths open: +0.3
        - Regular inside Big inside Tide: +0.2
        """
```

---

## Helper Functions

### Column Name Resolution

```python
def get_alligator_column_names_from_ctx_name(
    ctx_name: str
) -> tuple[str, str, str]:
    """
    Get jaw, teeth, lips column names for Alligator type.
    
    Args:
        ctx_name: "normal", "big", or "tide"
    
    Returns:
        (jaw_col, teeth_col, lips_col)
        
    Examples:
        "normal" -> ("jaw", "teeth", "lips")
        "big" -> ("bjaw", "bteeth", "blips")
        "tide" -> ("tjaw", "tteeth", "tlips")
    """
```

### Filtering Functions

```python
def filter_sig_is_in_ctx_teeth(
    df: pd.DataFrame,
    bs: str,
    ctx_name: str
) -> pd.DataFrame:
    """Filter bars where signal is in context Alligator teeth."""

def filter_sig_ctx_mouth_is_open_and_in_ctx_teeth(
    df: pd.DataFrame,
    bs: str,
    ctx_name: str
) -> pd.DataFrame:
    """Filter bars where mouth is open and signal in teeth."""

def filter_sig_ctx_mouth_is_open_and_in_ctx_lips(
    df: pd.DataFrame,
    bs: str,
    ctx_name: str
) -> pd.DataFrame:
    """Filter bars where mouth is open and signal in lips."""

def filter_sig_is_out_of_normal_mouth_sell(df: pd.DataFrame) -> pd.DataFrame:
    """Filter sell signals outside normal Alligator mouth."""

def filter_sig_is_out_of_normal_mouth_buy(df: pd.DataFrame) -> pd.DataFrame:
    """Filter buy signals outside normal Alligator mouth."""

def filter_sig_normal_mouth_is_open_sell(df: pd.DataFrame) -> pd.DataFrame:
    """Filter bars where normal mouth is open for sell."""

def filter_sig_normal_mouth_is_open_buy(df: pd.DataFrame) -> pd.DataFrame:
    """Filter bars where normal mouth is open for buy."""
```

---

## CLI Interface

```python
# jgtml/alligator_cli.py

def main():
    """
    Unified Alligator CLI.
    
    Arguments:
        -i, --instrument: Instrument symbol
        -t, --timeframe: Timeframe
        -d: Direction ("B" or "S")
        --type: Alligator type(s) - "regular", "big", "tide", "all"
        --generate-spec: Generate .jgtml-spec file
        --fresh: Force fresh data
        -q, --quiet: Suppress output
    
    Examples:
        # Single Alligator type
        python -m jgtml.alligator_cli -i SPX500 -t D1 -d S --type tide
        
        # All three types (convergence analysis)
        python -m jgtml.alligator_cli -i EUR/USD -t H4 -d B --type all
        
        # Generate spec file for agents
        python -m jgtml.alligator_cli -i GBPUSD -t D1 -d S --generate-spec
    """
```

---

## Output Format

### Console Output

```
╔═══════════════════════════════════════════════════════════════╗
║          ALLIGATOR CONVERGENCE ANALYSIS: EUR/USD H4           ║
╠═══════════════════════════════════════════════════════════════╣
║ Direction: BUY                                                ║
╠═══════════════════════════════════════════════════════════════╣
║ REGULAR (5-8-13)                                              ║
║   Jaw: 1.0850  Teeth: 1.0870  Lips: 1.0890                   ║
║   Mouth: FEEDING  Direction: BULLISH  Spread: 0.0040         ║
╠═══════════════════════════════════════════════════════════════╣
║ BIG (34-55-89)                                                ║
║   BJaw: 1.0750  BTeeth: 1.0800  BLips: 1.0850                ║
║   Mouth: FEEDING  Direction: BULLISH  Spread: 0.0100         ║
╠═══════════════════════════════════════════════════════════════╣
║ TIDE (144-233-377)                                            ║
║   TJaw: 1.0600  TTeeth: 1.0700  TLips: 1.0800                ║
║   Mouth: FEEDING  Direction: BULLISH  Spread: 0.0200         ║
╠═══════════════════════════════════════════════════════════════╣
║ CONVERGENCE SCORE: 0.95                                       ║
║ ALIGNED DIRECTION: BULLISH                                    ║
║ RECOMMENDATION: Strong buy alignment across all timeframes    ║
╚═══════════════════════════════════════════════════════════════╝
```

### .jgtml-spec Output

```json
{
  "instrument": "EUR/USD",
  "timeframe": "H4",
  "direction": "B",
  "analysis_timestamp": "2026-01-31T14:00:00Z",
  "regular": {
    "type": "normal",
    "jaw": 1.0850,
    "teeth": 1.0870,
    "lips": 1.0890,
    "mouth_state": "feeding",
    "direction": "bullish",
    "spread": 0.0040
  },
  "big": {
    "type": "big",
    "jaw": 1.0750,
    "teeth": 1.0800,
    "lips": 1.0850,
    "mouth_state": "feeding",
    "direction": "bullish",
    "spread": 0.0100
  },
  "tide": {
    "type": "tide",
    "jaw": 1.0600,
    "teeth": 1.0700,
    "lips": 1.0800,
    "mouth_state": "feeding",
    "direction": "bullish",
    "spread": 0.0200
  },
  "convergence_score": 0.95,
  "aligned_direction": "bullish",
  "recommendation": "Strong buy alignment across all timeframes"
}
```

---

## Dependencies

```python
import pandas as pd
import numpy as np
from enum import Enum
from typing import Dict, Tuple, Optional, List

from jgtpy import JGTCDS as cds
from jgtml import jtc
from JGTBalanceAnalyzer import (
    get_alligator_column_names_from_ctx_name,
    filter_sig_is_in_ctx_teeth,
    filter_sig_ctx_mouth_is_open_and_in_ctx_teeth,
    filter_sig_ctx_mouth_is_open_and_in_ctx_lips
)
from jgtutils.jgtconstants import (
    LOW, HIGH, FDBB, FDBS,
    JAW, TEETH, LIPS,
    BJAW, BTEETH, BLIPS,
    TJAW, TTEETH, TLIPS
)
```

---

## Quality Criteria

✅ **Three Scales**: Regular (5-8-13), Big (34-55-89), Tide (144-233-377)  
✅ **Mouth Detection**: Sleeping, Awakening, Feeding, Sated states  
✅ **Convergence Scoring**: 0.0-1.0 alignment metric  
✅ **Direction Analysis**: Bullish/Bearish/Neutral for each scale  
✅ **CLI Complete**: Single command for full analysis  
✅ **Spec Generation**: .jgtml-spec for agentic workflows

===============================================
FILE: app.specs.md
PATH: jgtml/rispecs/app.specs.md
URL:  https://jgtml.jgwill.com/rispecs/app.specs.md
===============================================

# JGTML Application Specification

> Master specification for the JGT Trading Signal Analysis Platform

**Specification Version**: 1.0  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: A comprehensive trading signal analysis platform that validates FDB signals within larger market structure, provides multi-Alligator confluence detection, and generates ML-ready features for predictive trading.

**Achievement Indicator**: Users can scan for FDB signals, validate them against HTF alignment, assess risk through multi-Alligator analysis, and generate datasets for machine learning.

**Value Proposition**: Transform raw CDS signals into validated, context-aware trading opportunities with clear entry/exit strategies and ML-ready feature sets.

---

## Application Overview

JGTML is a Python package that:
1. Scans for FDB (Fractal Divergent Bar) signals across instruments
2. Validates signals against Higher Timeframe (HTF) alignment
3. Provides Regular/Big/Tide Alligator confluence analysis
4. Generates cross-timeframe features (TTF) for ML
5. Creates meta lag features (MLF) for predictive models
6. Outputs matrix (MX) datasets for training

---

## Structural Tension

**Current Reality**: Raw FDB signals from CDS lack context about higher timeframe trends, Alligator states, and validation criteria.

**Desired State**: Each signal is enriched with HTF alignment, multi-Alligator confluence, risk assessment, and ML-ready features.

**Natural Progression**: JGTML builds upon jgtpy CDS data, adding validation layers and feature engineering for trading decisions.

---

## Architecture Overview

```
┌─────────────────────────────────────────────────────────────────┐
│                    JGTML Signal Analysis                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  CDS Data (jgtpy)                                               │
│       ↓                                                         │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │           FDB Scanner & Signal Validation                   ││
│  │  fdb_scanner_2508.py → SignalOrderingHelper.py              ││
│  └─────────────────────────────────────────────────────────────┘│
│       ↓                                                         │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │           Multi-Alligator Analysis                          ││
│  │  Regular (5-8-13) | Big (34-55-89) | Tide (144-233-377)     ││
│  │  TideAlligatorAnalysis.py → alligator_cli.py                ││
│  └─────────────────────────────────────────────────────────────┘│
│       ↓                                                         │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │           Feature Engineering                                ││
│  │  TTF (ttfcli) → MLF (mlfcli) → MX (mxcli)                   ││
│  └─────────────────────────────────────────────────────────────┘│
│       ↓                                                         │
│  ML-Ready Datasets / Trading Decisions                          │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘
```

---

## FDB Scanner

### Purpose
Scan for Fractal Divergent Bar signals with validation and quality scoring.

### Usage

```bash
# Scan for FDB signals
fdbscan -i EUR/USD -t H4

# Scan multiple instruments
fdbscan -i "EUR/USD,SPX500,XAU/USD" -t "H1,H4,D1"

# With HTF alignment check
fdbscan -i EUR/USD -t H4 --htf
```

### Signal Output

```python
@dataclass
class FDBSignal:
    instrument: str         # "EUR/USD"
    timeframe: str          # "H4"
    direction: str          # "B" (buy) or "S" (sell)
    date: datetime          # Signal bar datetime
    entry_price: float      # Recommended entry
    stop_loss: float        # Below/above fractal
    risk_pips: float        # Entry to stop distance
    quality_score: float    # 0.0 - 1.0 signal quality
    htf_aligned: bool       # HTF trend alignment
    alligator_state: str    # "mouth_open", "sleeping", etc.
```

---

## Multi-Alligator Analysis

### Three Alligator Periods

| Type | Periods | Use Case |
|------|---------|----------|
| Regular | 5-8-13 | Quick direction, day trading |
| Big | 34-55-89 | Swing trading, weekly cycles |
| Tide | 144-233-377 | Position trading, monthly trends |

### CLI Usage

```bash
# Single Alligator type
python -m jgtml.alligator_cli -i SPX500 -t D1 -d S --type tide

# All three types (convergence)
python -m jgtml.alligator_cli -i EUR/USD -t H4 -d B --type all

# Generate .jgtml-spec for agents
python -m jgtml.alligator_cli -i GBPUSD -t D1 -d S --generate-spec
```

### Convergence Analysis

```python
@dataclass
class AlligatorConvergence:
    instrument: str
    timeframe: str
    regular: AlligatorState  # 5-8-13 analysis
    big: AlligatorState      # 34-55-89 analysis
    tide: AlligatorState     # 144-233-377 analysis
    convergence_score: float # 0.0 - 1.0
    aligned_direction: Optional[str]  # "B", "S", or None
```

---

## Feature Engineering Pipeline

### TTF - Cross-Timeframe Features

**Purpose**: Add higher timeframe context to lower timeframe signals.

```bash
# Generate TTF features
ttfcli -i EUR/USD -t H4 --pattern mz

# All patterns
ttfcli --all
```

**Added Features**:
- HTF Alligator state (D1, W1)
- HTF trend direction
- Multi-timeframe confluence score

### MLF - Meta Lag Features

**Purpose**: Add lagged features for ML prediction.

```bash
# Generate MLF features
mlfcli -i EUR/USD -t H4
```

**Added Features**:
- Previous N bar values
- Rolling statistics
- Change rates

### MX - Matrix Generation

**Purpose**: Generate ML-ready datasets with targets.

```bash
# Generate MX data
mxcli -i EUR/USD -t H4 --fresh
```

**Output**: Dataset with features + target variable (signal outcome)

---

## CLI Tools

### jgtapp - Main Application

```bash
# Tide Alligator analysis (legacy wrapper)
jgtapp tide -i SPX500 -t D1 B

# Add entry order
jgtapp fxaddorder -i EUR/USD -n 0.1 -r 1.0950 -d B -x 1.0900

# Move stop to Alligator line
jgtapp fxmvstopgator -i EUR/USD -t H4 -tid TRADE_ID --lips
```

### jgtmlcli - Data Processing

```bash
# Process signals for instrument
jgtmlcli -i SPX500 -t D1 --full --fresh

# Generate all data layers
jgtmlcli -i EUR/USD -t H4 --ttf --mlf
```

---

## Type Definitions

```python
from typing import Optional, List, Dict, Any
from dataclasses import dataclass
from enum import Enum

class AlligatorType(Enum):
    REGULAR = "regular"   # 5-8-13
    BIG = "big"          # 34-55-89
    TIDE = "tide"        # 144-233-377

class AlligatorMouth(Enum):
    SLEEPING = "sleeping"
    AWAKENING = "awakening"
    FEEDING = "feeding"
    SATED = "sated"

@dataclass
class AlligatorState:
    type: AlligatorType
    mouth: AlligatorMouth
    direction: Optional[str]  # "B" or "S" or None
    jaw: float
    teeth: float
    lips: float
    spread: float  # Distance between lines
    
@dataclass
class SignalValidation:
    signal: FDBSignal
    htf_aligned: bool
    regular_mouth: AlligatorMouth
    big_aligned: bool
    tide_direction: Optional[str]
    risk_reward: float
    quality_score: float
    recommendation: str  # "TAKE", "SKIP", "WAIT"

def scan_fdb(
    instrument: str,
    timeframe: str,
    validate_htf: bool = True
) -> List[FDBSignal]: ...

def analyze_alligator(
    instrument: str,
    timeframe: str,
    direction: str,
    alligator_type: AlligatorType = AlligatorType.REGULAR
) -> AlligatorState: ...

def validate_signal(
    signal: FDBSignal,
    check_htf: bool = True,
    check_tide: bool = True
) -> SignalValidation: ...
```

---

## Creative Advancement Scenarios

### Scenario: FDB Signal Validation

**Desired Outcome**: Validate FDB buy signal against HTF trend

**Current Reality**: fdbscan detected buy signal on EUR/USD H4

**Natural Progression**:
1. FDB detected on H4 with bullish divergence
2. Check D1 Alligator: mouth open, lips > teeth > jaw
3. Check H4 Regular Alligator: mouth opening
4. Calculate risk: entry at 1.0950, stop at 1.0900 (50 pips)
5. Score signal: HTF aligned + Alligator confirming = 0.85

**Resolution**: Signal validated with "TAKE" recommendation

### Scenario: Multi-Alligator Confluence

**Desired Outcome**: Find instruments where all Alligators align

**Current Reality**: Need strongest trend opportunities

**Natural Progression**:
1. User runs: `python -m jgtml.alligator_cli -i SPX500 -t D1 -d B --type all`
2. Regular (5-8-13): Lips > Teeth > Jaw ✓
3. Big (34-55-89): Clear uptrend, mouth open ✓
4. Tide (144-233-377): Major uptrend ✓
5. Convergence score: 0.95

**Resolution**: Triple Alligator convergence signals strong trend

### Scenario: ML Feature Generation

**Desired Outcome**: Create dataset for FDB outcome prediction

**Current Reality**: Historical signals need feature engineering

**Natural Progression**:
1. Start with CDS: `cdscli --fresh`
2. Add TTF: `ttfcli --pattern mz`
3. Add MLF: `mlfcli`
4. Generate MX: `mxcli --fresh`
5. Output: CSV with features + target (signal profit/loss)

**Resolution**: ML-ready dataset for training signal predictor

---

## Module Structure

```
jgtml/
├── __init__.py                 # Package exports
├── jgtapp.py                   # Main application CLI
├── jgtmlcli.py                 # Data processing CLI
├── fdb_scanner_2508.py         # FDB scanning (current version)
├── fdb_scanner_2408.py         # FDB scanning (legacy)
├── SignalOrderingHelper.py     # Signal validation logic
├── TideAlligatorAnalysis.py    # Multi-period Alligator
├── AlligatorIllusionDetector.py # Alligator pattern detection
├── alligator_cli.py            # Unified Alligator CLI
├── ttfcli.py                   # TTF feature generation
├── ttfsvc.py                   # TTF service layer
├── mlfcli.py                   # MLF feature generation
├── mlfsvc.py                   # MLF service layer
├── mxcli.py                    # MX matrix generation
├── mxsvc.py                    # MX service layer
├── jtc.py                      # Target calculation
├── jplt.py                     # Plotting utilities
├── trading_orchestrator.py     # Trade lifecycle
└── experiments/                # Experimental features
```

---

## Integration with JGT Ecosystem

```
jgtpy (CDS signals)
    ↓ provides signal data
jgtml (this package) ← Analysis layer
    ↓ provides validated signals
jgt-data-server
    ↓ serves analysis via API
jgt-code
    ↓ displays in terminal
Trading Execution
```

---

## Quality Criteria

✅ **FDB Validation**: Multi-layer signal quality assessment  
✅ **HTF Alignment**: Higher timeframe trend confirmation  
✅ **Triple Alligator**: Regular/Big/Tide convergence analysis  
✅ **Feature Engineering**: TTF→MLF→MX pipeline complete  
✅ **CLI Complete**: All analysis available via command line  
✅ **ML Ready**: Output datasets for model training

===============================================
FILE: fdb-scanner.spec.md
PATH: jgtml/rispecs/fdb-scanner.spec.md
URL:  https://jgtml.jgwill.com/rispecs/fdb-scanner.spec.md
===============================================

# FDB Scanner Specification

> Fractal Divergent Bar Signal Detection Engine

**Specification Version**: 1.0  
**Module**: `jgtml/fdb_scanner_2508.py` (current), `jgtml/fdb_scanner_2408.py` (legacy)  
**CLI Entry**: `fdbscan`, `fdbscan2`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: An automated scanner that detects valid FDB (Fractal Divergent Bar) trading signals across multiple instruments and timeframes, validates them against Alligator state, calculates risk, and generates executable entry order scripts.

**Achievement Indicator**: Running `fdbscan -i EUR/USD -t H4` produces:
- List of valid FDB buy/sell signals
- Entry rate and stop rate with tick adjustments
- Risk calculation in pips
- Executable shell script for order placement
- JSON and Markdown signal files

**Value Proposition**: Transform manual chart scanning into automated signal detection with validation, risk calculation, and ready-to-execute trade scripts.

---

## Structural Tension

**Current Reality**: Trader manually scans charts looking for FDB patterns, calculates entry/stop levels, and creates orders by hand.

**Desired State**: Automated scanner runs continuously or on-demand, detecting signals, validating against Alligator state, and outputting executable trade commands.

**Natural Progression**: Scanner loads CDS data → detects FDB signals → validates Alligator mouth state → calculates risk → generates entry order script.

---

## Core Algorithm

### FDB Detection Logic

```python
def detect_fdb_signal(signal_bar: pd.Series) -> Optional[str]:
    """
    Detect FDB signal from completed bar.
    
    FDB Buy (fdb == 1):
        - Price makes new fractal low
        - AO does NOT make new low (bullish divergence)
        - Creates buy entry opportunity
        
    FDB Sell (fdb == -1):
        - Price makes new fractal high
        - AO does NOT make new high (bearish divergence)
        - Creates sell entry opportunity
    
    Returns:
        "B" for buy, "S" for sell, None for no signal
    """
    fdb_value = signal_bar['fdb']
    if fdb_value == 1:
        return "B"  # Buy signal
    elif fdb_value == -1:
        return "S"  # Sell signal
    return None
```

### Signal Validation Flow

```
1. Load CDS Data
   └── Use cache if valid (timeframe-based expiration)
   └── Generate fresh if cache invalid

2. Get Last Two Bars
   └── signal_bar = last completed bar (index -2)
   └── current_bar = current incomplete bar (index -1)

3. Check FDB Signal
   └── signal_bar['fdb'] == 1 (buy) or -1 (sell)

4. Validate Signal Not Broken
   └── For Buy: current_bar.high <= entry_rate AND current_bar.low >= stop_rate
   └── For Sell: current_bar.low >= entry_rate AND current_bar.high <= stop_rate

5. Validate Alligator State
   └── is_mouth_open(bar, bs) - Lips/Teeth/Jaw properly ordered
   └── is_bar_out_of_mouth(bar, bs) - Price outside Alligator lines

6. Calculate Entry/Stop with Tick Adjustment
   └── Add/subtract tick_shift for spread buffer

7. Calculate Risk
   └── risk_in_pips = |entry_rate - stop_rate| / pip_size

8. Generate Output
   └── Shell script for jgtnewsession
   └── JSON signal file
   └── Markdown summary
```

---

## Type Definitions

### Input Types

```python
from dataclasses import dataclass
from typing import Optional, List, Dict, Any
from datetime import datetime
import pandas as pd

# Instrument format: "EUR/USD", "SPX500", "XAU/USD"
Instrument = str

# Timeframe format: "m1", "m5", "m15", "m30", "H1", "H2", "H3", "H4", "H6", "H8", "D1", "W1", "M1"
Timeframe = str

# Buy/Sell direction
Direction = str  # "B" or "S"

# TLID timestamp string: "YYMMDDHHMMSS"
TLID = str
```

### Output Types

```python
@dataclass
class FDBSignalResult:
    """Result from FDB signal detection and validation"""
    instrument: Instrument
    timeframe: Timeframe
    direction: Direction           # "B" or "S"
    tlid_id: TLID                  # Unique signal identifier
    entry_rate: float              # Entry price with tick adjustment
    stop_rate: float               # Stop loss price with tick adjustment
    lots: float                    # Position size
    total_risk: float              # Total monetary risk
    unit_risk: float               # Risk per unit
    pips_risk: float               # Risk in pips
    sh: str                        # Executable shell script
    htfsig: Optional[str]          # Higher timeframe signals string

@dataclass
class HTFSignals:
    """Higher timeframe context signals"""
    zone1: str                     # HTF level 1 zone ("buy", "sell", "gray")
    zone2: str                     # HTF level 2 zone
    fade1: bool                    # HTF level 1 MFI fade
    fade2: bool                    # HTF level 2 MFI fade
    squat1: bool                   # HTF level 1 MFI squat
    squat2: bool                   # HTF level 2 MFI squat
    b4zlc1: int                    # Bars before ZLC at HTF level 1
    b4zlc2: int                    # Bars before ZLC at HTF level 2
```

---

## Function Specifications

### Main Entry Point

```python
def main() -> None:
    """
    FDB Scanner main entry point.
    
    CLI Arguments:
        -i, --instrument: Instrument(s) to scan (comma-separated or single)
        -t, --timeframe: Timeframe(s) to scan (comma-separated or single)
        -nc, --no-cache: Force fresh data, ignore cache
        --demo: Use demo broker connection
        -v, --verbose: Verbosity level (0-3)
        -q, --quiet: Suppress output
    
    Environment Variables:
        INSTRUMENTS: Override instrument list
        TIMEFRAMES: Override timeframe list
        LOTS: Position size (default: 1)
        JGT_CACHE: Cache directory path
    
    Output:
        - Console: Signal summaries
        - Files: 
          - rjgt/{instrument}_{timeframe}_{tlid}.sh - Individual signal scripts
          - rjgt/fdb_signals_out__{date}.sh - Combined script
          - data/jgt/signals/fdb_signals_out__{date}.json - All signals JSON
    """
```

### Cache Management

```python
def is_timeframe_cached_valid(
    df: pd.DataFrame, 
    timeframe: Timeframe,
    use_utc: bool = True,
    quiet: bool = True
) -> bool:
    """
    Check if cached data is still valid for the given timeframe.
    
    Validity is based on whether current time is within the timeframe period
    of the last bar in the cache.
    
    Args:
        df: Cached DataFrame with Date column or DatetimeIndex
        timeframe: Timeframe to validate ("m1" through "M1")
        use_utc: Use UTC time comparison
        quiet: Suppress debug output
    
    Returns:
        True if cache is still valid, False if refresh needed
    
    Timeframe Expiration Logic:
        m1: 1 minute after last bar
        m5: 5 minutes after last bar
        m15: 15 minutes after last bar
        m30: 30 minutes after last bar
        H1: 1 hour after last bar
        H2-H8: Next boundary (e.g., H4 at 00:00, 04:00, 08:00, 12:00, 16:00, 20:00)
        D1: 24 hours after last bar
        W1: 7 days after last bar
        M1: 30 days after last bar
    """

def generate_fresh_and_cache(
    instrument: Instrument,
    timeframe: Timeframe,
    quotescount: int = 300,
    cache_filepath: Optional[str] = None
) -> pd.DataFrame:
    """
    Generate fresh CDS data and cache it.
    
    Uses jgtpy.JGTCDSSvc.get() to fetch data, then saves to cache file.
    
    Args:
        instrument: Instrument symbol
        timeframe: Timeframe
        quotescount: Number of bars to fetch
        cache_filepath: Override cache file path
    
    Returns:
        Fresh CDS DataFrame
    """
```

### Signal Validation

```python
def is_mouth_open(bar: pd.Series, bs: Direction) -> bool:
    """
    Check if Alligator mouth is open in the given direction.
    
    For Buy (bs="B"):
        lips < teeth < jaw (Alligator pointing down, mouth open upward)
    
    For Sell (bs="S"):
        lips > teeth > jaw (Alligator pointing up, mouth open downward)
    
    Args:
        bar: Price bar with 'lips', 'teeth', 'jaw' columns
        bs: Direction to check ("B" or "S")
    
    Returns:
        True if mouth is open in the specified direction
    """

def is_bar_out_of_mouth(bar: pd.Series, bs: Direction) -> bool:
    """
    Check if price bar is outside the Alligator mouth.
    
    For Buy:
        bar.High < bar.Lips (price below all Alligator lines)
        AND mouth not open in opposite direction
    
    For Sell:
        bar.Low > bar.Lips (price above all Alligator lines)
        AND mouth not open in opposite direction
    
    Args:
        bar: Price bar with 'High', 'Low', 'lips' columns
        bs: Direction ("B" or "S")
    
    Returns:
        True if bar is out of the Alligator mouth
    """

def valid_gator(
    last_bar_completed: pd.Series,
    current_bar: pd.Series,
    bs: Direction
) -> bool:
    """
    Validate Alligator state for both signal bar and current bar.
    
    Both bars must pass:
        - is_mouth_open(bar, bs)
        - is_bar_out_of_mouth(bar, bs)
    
    Returns:
        True if both bars have valid Alligator state
    """
```

### Risk Calculation

```python
def calculate_entry_risk(
    instrument: Instrument,
    bs: Direction,
    entry_rate: float,
    stop_rate: float,
    position_size: float,
    tick_shift: int = 1,
    rounding_add: int = 2,
    timeframe: Optional[Timeframe] = None,
    quiet: bool = True,
    verbose_level: int = 0
) -> tuple[float, float, float]:
    """
    Calculate risk metrics for an entry order.
    
    Args:
        instrument: Trading instrument
        bs: Buy/Sell direction
        entry_rate: Entry price
        stop_rate: Stop loss price
        position_size: Lots/units
        tick_shift: Ticks to add/subtract for spread buffer
        rounding_add: Additional decimal places for rounding
        timeframe: Optional timeframe context
        quiet: Suppress output
        verbose_level: Debug verbosity
    
    Returns:
        Tuple of (total_risk, risk_per_unit, risk_in_pips)
    
    Calculation:
        pip_size = get_pips(instrument)  # e.g., 0.0001 for EUR/USD
        tick_size = pip_size / 10
        
        # Adjust for spread
        if bs == "B":
            entry_rate += tick_size * tick_shift
            stop_rate -= tick_size * tick_shift
        else:  # "S"
            entry_rate -= tick_size * tick_shift
            stop_rate += tick_size * tick_shift
        
        risk_per_unit = abs(entry_rate - stop_rate)
        total_risk = risk_per_unit * position_size
        risk_in_pips = risk_per_unit / pip_size
    """

def get_entry_stop_rate_ticked(
    instrument: Instrument,
    bs: Direction,
    entry_rate: float,
    stop_rate: float,
    tick_shift: int = 1,
    rounding_add: int = 2,
    timeframe: Optional[Timeframe] = None
) -> tuple[float, float]:
    """
    Adjust entry and stop rates with tick buffer.
    
    For Buy: entry += tick, stop -= tick
    For Sell: entry -= tick, stop += tick
    
    Returns:
        Tuple of (adjusted_entry_rate, adjusted_stop_rate)
    """
```

### Order Generation

```python
def create_fdb_entry_order(
    instrument: Instrument,
    signal_bar: pd.Series,
    current_bar: pd.Series,
    lots: float = 1,
    tick_shift: int = 2,
    quiet: bool = True,
    valid_gator_mouth_open_in_mouth: bool = False,
    validate_signal_out_of_mouth: bool = True,
    timeframe: Optional[Timeframe] = None,
    validation_timestamp: Optional[datetime] = None,
    verbose_level: int = 0,
    demo_flag: bool = True
) -> tuple[Optional[FDBSignalResult], str]:
    """
    Create entry order from FDB signal with full validation.
    
    Validation Steps:
        1. Check if signal_bar has FDB signal (fdb == 1 or -1)
        2. Check if signal not already broken by current_bar price action
        3. Optionally validate Alligator mouth state
        4. Calculate entry/stop with tick adjustment
        5. Calculate risk metrics
        6. Generate executable shell script
    
    Args:
        instrument: Trading instrument
        signal_bar: Last completed bar with FDB signal
        current_bar: Current incomplete bar
        lots: Position size
        tick_shift: Tick buffer for spread
        quiet: Suppress output
        valid_gator_mouth_open_in_mouth: Require Alligator mouth validation
        validate_signal_out_of_mouth: Require bar out of mouth validation
        timeframe: Timeframe for context
        validation_timestamp: Signal bar timestamp
        verbose_level: Debug verbosity
        demo_flag: Use demo broker
    
    Returns:
        Tuple of (FDBSignalResult or None, message string)
        Returns (None, msg) if signal invalid or validation fails
    """

def generate_entry_order_script(
    lots: float,
    entry_rate: float,
    stop_rate: float,
    instrument: Instrument,
    buysell: Direction,
    tlid_id: Optional[TLID] = None,
    timeframe: Optional[Timeframe] = None,
    validation_timestamp_str: str = "",
    demo_flag: bool = True,
    total_risk: Optional[float] = None,
    risk_per_unit: Optional[float] = None,
    risk_in_pips: Optional[float] = None,
    extra_scripting_output: Optional[str] = None
) -> str:
    """
    Generate executable shell script for entry order.
    
    Output Format:
        ```sh
        ### --- COPY FROM HERE ---
        demo_arg="--demo"
        # FDB Buy Entry EUR/USD H4 - bts/now:2026-01-31 14:00/2026-01-31 14:05
        risk_in_pips=25.5
        instrument="EUR/USD";timeframe="H4";bs="B"
        tlid_id=260131140500;lots=1
        entry_rate=1.0950;stop_rate=1.0900
        jgtnewsession $tlid_id $instrument $timeframe $entry_rate $stop_rate $bs $lots $demo_arg
        zone=B-B-S-N-N;fade1=False;squat1=False;b4zlc1=5
        ### ---- COPY TO HERE ---
        ```
    """
```

---

## HTF Context Gathering

```python
def _get_htf_signal(timeframe: Timeframe) -> Optional[str]:
    """
    Gather higher timeframe context signals.
    
    For each scanned timeframe, looks up 1 and 2 levels higher to get:
        - Zone signal (buy/sell/gray)
        - MFI fade status
        - MFI squat status
        - Bars before ZLC
    
    Example for H1:
        HTF level 1 = H4
        HTF level 2 = D1
    
    Returns:
        String like "zone1=buy;fade1=False;squat1=False;b4zlc1=5;zone2=sell;..."
    """

def expand_timeframe_list(timeframes_to_parse: List[Timeframe]) -> None:
    """
    Expand timeframe list to include necessary HTF levels.
    
    If scanning m5, also need m15 for context
    If scanning m1, need m5 and m15
    
    Modifies list in place to add required timeframes.
    """
```

---

## Output Serialization

```python
from JGTOutputHelper import (
    serialize_signal_to_json_file,
    serialize_signal_to_markdown_file_from_json_file
)

# Signal saved to: data/jgt/signals/{instrument}_{timeframe}_{tlid}.json
# Markdown saved to: data/jgt/signals/{instrument}_{timeframe}_{tlid}.md
# Combined signals: data/jgt/signals/fdb_signals_out__{date}.json
# Combined script: rjgt/fdb_signals_out__{date}.sh
```

---

## CLI Usage Examples

```bash
# Scan single instrument/timeframe
fdbscan -i EUR/USD -t H4

# Scan multiple instruments
fdbscan -i "EUR/USD,SPX500,XAU/USD" -t "H1,H4,D1"

# Force fresh data (ignore cache)
fdbscan -i EUR/USD -t H4 --no-cache

# Verbose output
fdbscan -i EUR/USD -t H4 -v -v -v

# Demo mode (default)
fdbscan -i EUR/USD -t H4 --demo

# Real trading mode
fdbscan -i EUR/USD -t H4 --real

# Using environment variables
INSTRUMENTS="EUR/USD,GBP/USD" TIMEFRAMES="H1,H4" LOTS=0.1 fdbscan
```

---

## Creative Advancement Scenario

### Scenario: Morning FDB Scan

**Desired Outcome**: Trader discovers valid FDB signals at start of trading day

**Current Reality**: Markets just opened, need to find opportunities

**Natural Progression**:
1. Trader runs: `fdbscan -i "EUR/USD,GBP/USD,SPX500" -t "H1,H4"`
2. Scanner loads CDS data for each instrument/timeframe
3. For EUR/USD H4:
   - Last bar has `fdb = 1` (buy signal)
   - Current bar: High < entry_rate (signal not broken)
   - Alligator: Mouth open downward, price below lips
   - HTF D1: Zone is buy, no fade/squat
4. Signal validated, output generated:
   ```
   a=Scanning;i=EUR/USD;t=H4;vtlid=2601311400
   # FDB Buy Entry EUR/USD H4
   risk_in_pips=25.5
   . rjgt/EUR-USD_H4_260131140500.sh
   ```
5. Trader reviews and executes: `. rjgt/EUR-USD_H4_260131140500.sh`

**Resolution**: Validated trading signal with risk calculated and order ready

---

## Dependencies

```python
# External
import pandas as pd
from datetime import datetime, timedelta
import json
import os
import signal
import atexit

# JGT Packages
from jgtpy import JGTCDSSvc as svc  # CDS data service
from jgtutils.jgtconstants import LOW, HIGH, FDB, ZONE_SIGNAL, MFI_FADE, MFI_SQUAT
from jgtutils.coltypehelper import DTYPE_DEFINITIONS
from jgtutils import jgtcommon  # CLI argument parsing
from jgtutils.iprops import get_pips  # Pip size lookup

# JGTML Internal
import tlid  # Timestamp ID generation
from xhelper import count_bars_before_zero_line_cross
from SOHelper import get_bar_at_index, get_last_two_bars
from SignalOrderingHelper import create_fdb_entry_order
from JGTOutputHelper import serialize_signal_to_json_file
import JGTBalanceAnalyzer as ba
from mlconstants import *  # Alligator column names
```

---

## Quality Criteria

✅ **Signal Validation**: Multi-layer validation (FDB → broken → Alligator → HTF)  
✅ **Risk Calculation**: Tick-adjusted entry/stop with pips risk  
✅ **Cache Management**: Timeframe-aware cache expiration  
✅ **Output Formats**: Shell script, JSON, Markdown  
✅ **HTF Context**: Higher timeframe zone/fade/squat/b4zlc  
✅ **Graceful Cleanup**: Signal handlers for SIGINT/SIGTERM  
✅ **Tracing Support**: JGTTracer integration for observability

===============================================
FILE: jgtapp.spec.md
PATH: jgtml/rispecs/jgtapp.spec.md
URL:  https://jgtml.jgwill.com/rispecs/jgtapp.spec.md
===============================================

# jgtapp (Trading Application CLI) Specification

> Unified CLI for Trading Operations and Data Workflows

**Specification Version**: 1.0  
**Module**: `jgtml/jgtapp.py`  
**CLI Command**: `jgtapp`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Executed trading operations (orders, stop management) and data pipeline commands through a unified interface.

**Achievement Indicator**: Running `jgtapp fxaddorder -i EUR/USD -n 1 -r 1.0950 -d B -x 1.0900 --demo` produces:
- Entry order placed on broker
- Order confirmation with ID

**Value Proposition**: Single command interface for entire trading workflow - from data refresh to order execution to trade management.

---

## Structural Tension

**Current Reality**: Multiple CLI tools across packages require remembering different commands.

**Desired State**: Unified `jgtapp` command with subcommands for all trading operations.

**Natural Progression**: Parse subcommand → Execute corresponding function → Output JSONL status.

---

## CLI Architecture

```
jgtapp
├── Trading Operations
│   ├── fxaddorder (add)     - Create entry orders
│   ├── fxrmorder (rm)       - Remove pending orders
│   ├── fxrmtrade (close)    - Close open trades
│   ├── fxtr                 - Get trade/order details
│   ├── fxmvstop (mv)        - Move stop loss
│   ├── fxmvstopgator        - Move stop to Alligator line
│   └── fxmvstopfdb (fdb)    - Move stop to FDB signal
│
├── Data Operations
│   ├── pds                  - Refresh price data
│   ├── ids                  - Refresh indicators
│   ├── cds                  - Refresh signals
│   ├── ads                  - Generate charts
│   ├── ttf                  - Cross-timeframe features
│   ├── mlf                  - Meta lag features
│   └── mx                   - ML matrix
│
├── Workflows
│   ├── ttfmxwf              - Full TTF→MX pipeline
│   ├── ttfwf                - TTF preparation workflow
│   ├── ocds                 - Old CDS refresh
│   └── w                    - Wait for timeframe
│
├── Analysis
│   ├── tide                 - Alligator analysis
│   └── entryvalidate        - Validate pending entry
```

---

## Trading Operations

### fxaddorder - Create Entry Order

```python
def fxaddorder(
    instrument: str,
    lots: str,
    rate: str,
    buysell: str,
    stop: str,
    demo: bool = False,
    flag_pips: bool = False
) -> None:
    """
    Create entry stop order on broker.
    
    Wraps: fxaddorder CLI from jgtfxcon
    
    Args:
        instrument: Trading pair (EUR/USD)
        lots: Position size in lots
        rate: Entry price
        buysell: Direction (B/S)
        stop: Stop loss price (or pips if flag_pips)
        demo: Use demo account
        flag_pips: Interpret stop as pips distance
    """
```

### fxrmorder - Remove Order

```python
def fxrmorder(orderid: str, demo: bool = False) -> None:
    """
    Remove pending entry order.
    
    Used when:
        - Signal invalidated
        - Stop level hit before entry
        - Trading plan changed
    """
```

### fxmvstopgator - Stop to Alligator Line

```python
def fxmvstopgator(
    i: str,                    # Instrument
    t: str,                    # Timeframe
    tradeid: str,              # Trade ID
    lips: bool = True,         # Use lips (default)
    teeth: bool = False,       # Use teeth
    jaw: bool = False,         # Use jaw
    demo: bool = False,
    skip_trade_data_update: bool = False,
    loop_action: bool = False  # Recursive on timeframe
) -> None:
    """
    Move stop to Alligator balance line.
    
    Algorithm:
        1. Get trade data (direction, current stop)
        2. Update IDS for instrument/timeframe
        3. Get current bar's Alligator values
        4. Select line (lips/teeth/jaw)
        5. Move stop to selected line value
        6. If loop_action: wait for next TF, recurse
    
    Use case:
        - Trailing stop on Alligator lines
        - Automated stop management during trade
    """
```

### fxmvstopfdb - Stop to FDB Signal

```python
def fxmvstopfdb(
    i: str,
    t: str,
    tradeid: str,
    demo: bool = False,
    close: bool = False,
    lips: bool = False,
    teeth: bool = False,
    jaw: bool = False,
    not_if_stop_closer: bool = True
) -> None:
    """
    Move stop to FDB signal level.
    
    Algorithm:
        1. Get trade data
        2. Update IDS, calculate FDB column
        3. Get last completed bar
        4. If opposite FDB signal:
           - Calculate stop from bar high/low
           - If stop already hit: close trade
           - Else: move stop to FDB level
           - Set moved_to_fdb flag
        5. If no FDB signal and lips/teeth/jaw:
           - Check if previously moved to FDB
           - If not: fallback to Alligator line
    
    Flags stored: .jgt/fdb_moved_flag_{tradeid}.json
    """
```

---

## Data Operations

### Wrappers for Pipeline CLIs

```python
def pds(instrument: str, timeframe: str, use_full: bool = True) -> None:
    """Refresh PDS (price data) via pdscli."""

def ids(instrument: str, timeframe: str, use_full: bool = False, use_fresh: bool = True) -> None:
    """Refresh IDS (indicators) via idscli."""

def cds(instrument: str, timeframe: str, use_fresh: bool = False, use_full: bool = True) -> None:
    """Refresh CDS (signals) via cdscli."""

def ads(instrument: str, timeframe: str, use_fresh: bool = False, tc: bool = True, pov: bool = False) -> None:
    """Generate ADS charts via adscli."""

def ttf(instrument: str, timeframe: str, pn: str = "ttf", use_fresh: bool = False, use_full: bool = True) -> None:
    """Generate TTF (cross-timeframe features) via ttfcli."""

def mlf(instrument: str, timeframe: str, pn: str = "ttf", total_lagging_periods: int = 5, ...) -> None:
    """Generate MLF (meta lag features) via mlfcli."""

def mx(instrument: str, timeframe: str, use_fresh: bool = False) -> None:
    """Generate MX (ML matrix) via mxcli."""
```

---

## Workflow Operations

### ttfmxwf - Full ML Preparation

```python
def ttfmxwf(instrument: str, use_fresh: bool = False) -> None:
    """
    Complete ML data preparation workflow.
    
    For each timeframe [MN, W1, D1, H4]:
        1. Refresh CDS
        2. Generate TTF (except MN)
        3. Generate MX (except MN, W1)
    """
```

### w - Timeframe Wait

```python
def w(
    timeframe: str,
    script_to_run: str = None,
    exit_on_timeframe: bool = False
) -> None:
    """
    Wait for timeframe, then execute or exit.
    
    Wraps: tfw CLI from jgtutils
    
    Used by:
        - fxmvstopgator loop_action
        - Trading campaign automation
    """
```

---

## Analysis Operations

### tide - Alligator Analysis

```python
def tide(
    instrument: str,
    timeframe: str,
    buysell: str,
    type: str = 'tide',
    quiet: bool = False
) -> None:
    """
    Unified Alligator analysis.
    
    Types:
        - tide: 144-233-377 periods (macro)
        - big: 34-55-89 periods (intermediate)
        - regular: 5-8-13 periods (quick)
        - all: All three analyses
    
    Wraps: alligator_cli unified CLI
    """
```

### entryvalidate - Validate Pending Entry

```python
def entryvalidate(orderid: str, timeframe: str, demo: bool = False) -> None:
    """
    Validate pending entry order against current price.
    
    Algorithm:
        1. Get order data (instrument, direction, stop)
        2. Check if order became a trade
        3. If still pending:
           - Get current bar close
           - If stop hit: remove order
    
    Used in campaign automation to clean invalid entries.
    """
```

---

## Helper Functions

### Trade Data Access

```python
def _get_trade_data(tradeid: str, demo: bool, fresh: bool = True) -> dict:
    """Load trade details from cached or fresh data."""

def _get_order_data(orderid: str, demo: bool) -> dict:
    """Load order details from cached or fresh data."""

def _get_instrument_from_orderid(orderid: str, demo: bool) -> str:
    """Extract instrument from order data."""

def order_became_a_trade(orderid: str, demo: bool) -> bool:
    """Check if pending order has been filled."""
```

### IDS Update

```python
def _get_ids_updated(i: str, t: str, skip_generating: bool = False) -> pd.DataFrame:
    """
    Get fresh IDS with FDB column.
    
    1. Run ids() to refresh
    2. Read IDS CSV
    3. Add FDB column via _ids_add_fdb_column_logics_v2
    """
```

---

## JSONL Output

```python
def print_jsonl_message(
    msg: str,
    extra_dict: dict = None,
    scope: str = None
) -> None:
    """
    Print structured JSONL message.
    
    Format:
        {"message": "...", "trade_id": "123", "scope": "jgtapp::fxtr"}
    """
```

---

## CLI Constants Used

```python
from jgtutils.jgtcliconstants import (
    CLI_FXADDORDER_PROG_NAME,    # fxaddorder
    CLI_FXMVSTOP_PROG_NAME,      # fxmvstop
    CLI_FXRMORDER_PROG_NAME,     # fxrmorder
    CLI_FXRMTRADE_PROG_NAME,     # fxrmtrade
    CLI_FXTR_PROG_NAME,          # fxtr
    PDSCLI_PROG_NAME             # jgtfxcli
)

from jgtpy.jgtpyconstants import (
    IDSCLI_PROG_NAME,   # idscli
    CDSCLI_PROG_NAME,   # cdscli
    ADSCLI_PROG_NAME,   # adscli
    JGTCLI_PROG_NAME    # jgtcli
)

from mlcliconstants import (
    MLFCLI_PROG_NAME,   # mlfcli
    TTFCLI_PROG_NAME,   # ttfcli
    MXCLI_PROG_NAME     # mxcli
)
```

---

## Integration Points

### jgtfxcon (Broker Layer)
- fxaddorder, fxrmorder, fxtr, fxmvstop → Direct CLI calls
- Transaction data stored in `data/jgt/fxtransact*.json`

### jgtpy (Data Layer)
- ids, cds, ads → Refresh data pipeline
- Read IDS for Alligator values

### jgtutils (Utilities)
- tfw for timeframe waiting
- FXTransact for trade data structures

### jgt-data-server
- Could expose jgtapp functions via API

### jgt-code
- Uses jgtapp for trade execution
- Medicine Wheel North direction

---

## Dependencies

```python
import subprocess
import json
from jgtutils import jgtcommon
from jgtutils.FXTransact import FXTransactWrapper, FXTrade, FXTrades
from jgtpy.JGTIDS import _ids_add_fdb_column_logics_v2
from jgtpy import jgtapyhelper as th
from SOHelper import get_bar_at_index
from alligator_cli import main as alligator_main
```

---

## Quality Criteria

✅ **Unified Interface**: All trading operations in one CLI  
✅ **Wrapper Pattern**: Delegates to specialized CLIs  
✅ **JSONL Output**: Structured logging for automation  
✅ **Demo/Real**: Account type switching  
✅ **FDB Integration**: Advanced stop management with signals  
✅ **Loop Actions**: Recursive execution on timeframes  
✅ **Campaign State**: Flag files for FDB moved tracking

===============================================
FILE: mlf.spec.md
PATH: jgtml/rispecs/mlf.spec.md
URL:  https://jgtml.jgwill.com/rispecs/mlf.spec.md
===============================================

# MLF (Meta Lag Features) Specification

> Lagged Features for Machine Learning

**Specification Version**: 1.0  
**Modules**: `jgtml/mlfcli.py`, `jgtml/mlfsvc.py`, `jgtml/realityhelper.py`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: ML-ready feature datasets with time-lagged versions of indicators, enabling models to learn from temporal patterns (how did indicators behave over the last N bars before a signal?).

**Achievement Indicator**: Running `mlfcli -i EUR/USD -t H1 -pn mz -lp 1 -tlp 5` produces:
- CSV file with original TTF features
- Lagged columns: `ao_lag_1`, `ao_lag_2`, ..., `ao_lag_5` for each feature
- Feature count multiplied by (1 + lag_periods)

**Value Proposition**: ML models need temporal context. MLF transforms static snapshots into sequences that reveal momentum, acceleration, and pattern evolution.

---

## Structural Tension

**Current Reality**: TTF data shows current state with HTF context but no temporal memory.

**Desired State**: Each row contains current values plus lagged history, enabling sequence-aware ML training.

**Natural Progression**: TTF → Add lag columns → Clean NaN → Save MLF CSV.

---

## Data Pipeline Position

```
PDS → IDS → CDS → TTF → [MLF] → MX
                          ↑
                    Current stage
```

**Dependencies**: Requires TTF data for the specified pattern name.

---

## Core Function: generate_mlf_feature_pattern

```python
def generate_mlf_feature_pattern(
    i: str,                              # Instrument symbol
    t: str,                              # Timeframe
    lag_period: int = 1,                 # Lag step size (usually 1)
    total_lagging_periods: int = 5,      # Number of lag periods
    dropna: bool = True,                 # Drop rows with NaN
    use_full: bool = True,               # Full vs current data
    columns_to_keep: List[str] = None,   # Columns to retain
    columns_to_drop: List[str] = None,   # Columns to remove
    drop_bid_ask: bool = False,          # Remove bid/ask columns
    force_refresh: bool = False,         # Force regenerate TTF
    quiet: bool = True,                  # Suppress output
    pn: str = "ttf",                     # Pattern name
    out_lag_midfix_str: str = '_lag_',   # Lag column naming
    just_keep_lagging_columns: bool = False,  # Only keep lag columns
    save_to_csv: bool = True,            # Write output file
    args: argparse.Namespace = None
) -> pd.DataFrame:
    """
    Generate MLF feature dataset with lagged columns.
    
    Algorithm:
        1. Read pattern column metadata from TTF
        2. Load TTF DataFrame for pattern
        3. Clean DataFrame (drop columns, remove bid/ask)
        4. For each column in columns_list_from_higher_tf:
           - Create lag_1, lag_2, ..., lag_N columns
           - Each lag_k = column shifted by k periods
        5. Write MLF pattern metadata
        6. Save to CSV
    
    Returns:
        DataFrame with original + lagged features
    
    Output Path:
        $JGTPY_DATA/mlf/{instrument}_{timeframe}_{patternname}.csv
    """
```

---

## Lagging Algorithm

### Column Creation

```python
def add_lagging_columns(
    df: pd.DataFrame,
    columns_list: List[str],
    lag_period: int = 1,
    total_lagging_periods: int = 5,
    out_lag_midfix_str: str = '_lag_'
) -> pd.DataFrame:
    """
    Add lagged versions of specified columns.
    
    For each column in columns_list:
        for k in range(1, total_lagging_periods + 1):
            lag_step = k * lag_period
            new_col = f"{column}{out_lag_midfix_str}{k}"
            df[new_col] = df[column].shift(lag_step)
    
    Example (lag_period=1, total_lagging_periods=3):
        ao -> ao_lag_1, ao_lag_2, ao_lag_3
        
    Example with lag_period=2, total_lagging_periods=3:
        ao -> ao_lag_1 (shift 2), ao_lag_2 (shift 4), ao_lag_3 (shift 6)
    """
```

### Lagging Example

Original data:
```
Date        ao      zone_sig
2026-01-30  0.0025  B
2026-01-31  0.0030  B
2026-02-01  0.0028  S
2026-02-02  0.0035  B
2026-02-03  0.0040  B
```

After MLF with lag_period=1, total_lagging_periods=2:
```
Date        ao      ao_lag_1  ao_lag_2  zone_sig  zone_sig_lag_1  zone_sig_lag_2
2026-01-30  0.0025  NaN       NaN       B         NaN             NaN
2026-01-31  0.0030  0.0025    NaN       B         B               NaN
2026-02-01  0.0028  0.0030    0.0025    S         B               B
2026-02-02  0.0035  0.0028    0.0030    B         S               B
2026-02-03  0.0040  0.0035    0.0028    B         B               S
```

---

## CLI Interface

```python
# jgtml/mlfcli.py

def main():
    """
    MLF CLI Entry Point.
    
    Arguments:
        -i, --instrument: Instrument symbol (required)
        -t, --timeframe: Timeframe (required)
        -pn, --patternname: Pattern name (default: "ttf")
        -lp, --lag_period: Lag step size (default: 1)
        -tlp, --total_lagging_periods: Number of lags (default: 5)
        --fresh: Force refresh TTF data
        --full: Use full historical data
        -ctk, --columns_to_keep: Columns to retain
        -ctd, --columns_to_drop: Columns to remove
        --rmbidask: Remove bid/ask columns
        --dropna-volume: Drop zero-volume rows
    
    Examples:
        # Default MLF with 5 lag periods
        mlfcli -i EUR/USD -t H1 -pn mz
        
        # Custom lag configuration
        mlfcli -i EUR/USD -t H1 -pn mz -lp 1 -tlp 10
        
        # Specific columns only
        mlfcli -i SPX500 -t D1 -pn mz -ctk ao zone_sig mfi_sig
        
        # Fresh data regeneration
        mlfcli -i GBPUSD -t H4 --fresh
    """
```

---

## Programmatic API

```python
def generate_mlf_for_pattern(
    instrument: str,
    timeframe: str,
    patternname: str = "ttf",
    lag_period: int = 1,
    total_lagging_periods: int = 5,
    use_full: bool = True,
    use_fresh: bool = False,
    columns_to_keep: List[str] = None,
    columns_to_drop: List[str] = None,
    drop_bid_ask: bool = False,
    args: argparse.Namespace = None
) -> pd.DataFrame:
    """
    Programmatic wrapper for MLF generation.
    
    Includes JGTTracer integration for observability.
    """
```

---

## Service Layer

```python
class MLFService:
    """
    MLF Service for programmatic access.
    """
    
    def generate_features(self, request: MLFRequest) -> pd.DataFrame:
        """
        Generate MLF features from request object.
        
        If TTF pattern not found, suggests running:
            ttfcli -i {instrument} -t {timeframe} -pn {pattern}
        """
    
    def create_request(
        self,
        instrument: str,
        timeframe: str,
        pn: str,
        lag_period: int = 1,
        total_lagging_periods: int = 5,
        use_full: bool = True,
        force_refresh: bool = False,
        dropna: bool = True,
        columns_to_keep: List[str] = None,
        columns_to_drop: List[str] = None,
        drop_bid_ask: bool = True
    ) -> MLFRequest:
        """Create MLFRequest object for service consumption."""
```

---

## Output Schema

### Feature Multiplication

```
Original columns: N
Lag periods: L
Total output columns: N * (1 + L)

Example:
- 10 base features
- 5 lag periods
- 10 * 6 = 60 total features
```

### Column Naming Pattern

```python
# Original column: {name}
# Lagged columns: {name}_lag_{k}

# Example for 'ao' with 5 lags:
ao           # Current value
ao_lag_1     # 1 bar ago
ao_lag_2     # 2 bars ago
ao_lag_3     # 3 bars ago
ao_lag_4     # 4 bars ago
ao_lag_5     # 5 bars ago
```

---

## File Locations

```python
def get_mlf_outfile_fullpath(
    i: str,
    t: str,
    use_full: bool,
    suffix: str = "",
    ns: str = "mlf"
) -> str:
    """
    Get output file path for MLF CSV.
    
    Path Construction:
        base_dir = $JGTPY_DATA/mlf (current) or $JGTPY_DATA_FULL/mlf (full)
        filename = {instrument}_{timeframe}_{suffix}.csv
        
    Examples:
        -> /src/jgtml/data/current/mlf/EUR-USD_H1_mz.csv
        -> /src/jgtml/data/current/mlf/EUR-USD_H1_mz_columns.json
    """
```

---

## Fallback Logic

```python
# If TTF not found, attempt auto-generation:
try:
    df = generate_mlf_feature_pattern(...)
except Exception:
    print("TTF not found, generating...")
    from jgtapp import ttf
    ttf(instrument, timeframe, pn=patternname, use_fresh=fresh)
    df = generate_mlf_feature_pattern(...)  # Retry
```

---

## DataFrame Cleaning

```python
def __clean_dataframe(
    df: pd.DataFrame,
    columns_to_keep: List[str] = None,
    columns_to_drop: List[str] = None,
    drop_bid_ask: bool = False,
    dropna: bool = True
) -> pd.DataFrame:
    """
    Clean DataFrame before adding lags.
    
    Operations:
        1. dropna() if requested
        2. Remove columns_to_drop
        3. If drop_bid_ask: remove BidOpen, BidHigh, BidLow, BidClose,
           AskOpen, AskHigh, AskLow, AskClose
    """
```

---

## Dependencies

```python
import pandas as pd
import argparse
from jgtutils import jgtcommon
from jgtml.ptottf import read_ttf_csv
from jgtml.mldatahelper import (
    pndata__read_new_pattern_columns_list_with_htf,
    write_mlf_pattern_lagging_columns_list,
    _get_lagging_columns_list
)
from jgtml.anhelper import add_lagging_columns
from jgtml.realityhelper import generate_mlf_feature_pattern
```

---

## Quality Criteria

✅ **Temporal Context**: Models can see N periods of history  
✅ **Configurable Lags**: Adjustable lag_period and total_lagging_periods  
✅ **Pattern Aware**: Uses TTF pattern column definitions  
✅ **Clean Data**: NaN rows dropped, bid/ask removal option  
✅ **Auto-fallback**: Generates TTF if missing  
✅ **Metadata**: Column definitions saved for MX consumption

---

## Downstream Usage

```python
# MLF is consumed by MX for final ML matrix:
from jgtml.mxcli import create_mx_dataset

mx_df = create_mx_dataset(
    "EUR/USD", "H1",
    patternname="mz",
    target_column="fdb"
)
```

===============================================
FILE: mx.spec.md
PATH: jgtml/rispecs/mx.spec.md
URL:  https://jgtml.jgwill.com/rispecs/mx.spec.md
===============================================

# MX (ML Matrix) Specification

> Final ML-Ready Feature Matrix with Target Calculation

**Specification Version**: 1.0  
**Modules**: `jgtml/mxcli.py`, `jgtml/mxsvc.py`, `jgtml/jtc.py`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Complete ML-ready dataset with features and calculated target variable showing actual profit/loss outcomes for each FDB signal.

**Achievement Indicator**: Running `mxcli -i EUR/USD -t H1` produces:
- MX CSV with all features plus calculated `target` column
- Target shows actual pips gained/lost if signal was taken
- Ready for ML model training/evaluation

**Value Proposition**: Training data that teaches models the true outcome of each signal - not just whether it was a valid signal, but how much money it made or lost.

---

## Structural Tension

**Current Reality**: MLF data with features but no labels - we don't know which signals were profitable.

**Desired State**: Dataset with calculated target variable showing actual P&L for each signal over a forward window.

**Natural Progression**: MLF → Apply target calculation → Forward-looking P&L → Save MX CSV.

---

## Data Pipeline Position

```
PDS → IDS → CDS → TTF → MLF → [MX]
                                ↑
                          Current stage
```

**Dependencies**: Can use MLF data or direct CDS data with TTF enrichment.

---

## Core Function: pto_target_calculation

```python
def pto_target_calculation(
    i: str,                              # Instrument symbol
    t: str,                              # Timeframe
    crop_start_dt: str = None,           # Start date for data crop
    crop_end_dt: str = None,             # End date for data crop
    tlid_tag: str = None,                # Timestamp ID tag
    WINDOW_MIN: int = 1,                 # Minimum forward window
    WINDOW_MAX: int = 150,               # Maximum forward window (bars)
    output_report_dir: str = None,       # Report output directory
    pto_vec_fdb_ao_vector_window_flag: bool = True,  # Calculate AO vectors
    drop_calc_col: bool = True,          # Drop intermediate columns
    write_reporting: bool = True,        # Write reports
    selected_columns_to_keep: List[str] = None,  # Feature selection
    save_outputs: bool = True,           # Save to CSV
    keep_bid_ask: bool = True,           # Keep bid/ask columns
    use_fresh: bool = True,              # Force fresh data
    regenerate_cds: bool = False,        # Regenerate CDS first
    gator_oscillator_flag: bool = False, # Include Gator Oscillator
    mfi_flag: bool = True,               # Include MFI
    balligator_flag: bool = False,       # Include Big Alligator
    balligator_period_jaws: int = 89,    # Big Alligator jaw period
    talligator_flag: bool = False,       # Include Tide Alligator
    talligator_period_jaws: int = 377,   # Tide Alligator jaw period
    use_ttf: bool = True,                # Use TTF enrichment
    pn: str = "ttf",                     # Pattern name
    quiet: bool = True
) -> Tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]:
    """
    Calculate target variable for FDB signals.
    
    Returns:
        (df_result_tmx, sel1, sel2):
        - df_result_tmx: Full MX matrix with all features and target
        - sel1: Selected columns subset (_sel)
        - sel2: Minimal target-only subset (_tnd)
    
    Output Paths:
        $JGTPY_DATA_FULL/targets/mx/{instrument}_{timeframe}.csv
        $JGTPY_DATA_FULL/targets/mx/{instrument}_{timeframe}_sel.csv
        $JGTPY_DATA_FULL/targets/mx/{instrument}_{timeframe}_tnd.csv
    """
```

---

## Target Calculation Algorithm

### Calculate Target for Each FDB Signal

```python
def calculate_target_variable_min_max(
    dfsrc: pd.DataFrame,
    crop_last_dt: str = None,
    crop_start_dt: str = None,
    WINDOW_MIN: int = 1,
    WINDOW_MAX: int = 150,
    set_index: bool = True,
    rounder: int = 4,
    pipsize: float = -1,
    target_colname: str = ""
) -> pd.DataFrame:
    """
    Calculate P&L target for each FDB signal.
    
    Algorithm:
        For each row i with FDB Sell signal (fdbs=1):
            tmax = min(Close[i:i+WINDOW_MAX])  # Best exit price
            tmin = max(Close[i:i+WINDOW_MAX])  # Worst case
            
            if High[i] < tmin:  # Stop hit before profit
                target = -1 * (High - Low)     # Negative (loss)
            else:
                target = Low - tmax            # Positive (profit)
        
        For each row i with FDB Buy signal (fdbb=1):
            tmax = max(Close[i:i+WINDOW_MAX])  # Best exit price
            tmin = min(Close[i:i+WINDOW_MAX])  # Worst case
            
            if Low[i] > tmin:  # Stop hit before profit
                target = -1 * (High - Low)     # Negative (loss)
            else:
                target = tmax - High           # Positive (profit)
    
    Intermediate Columns:
        tmax: Best possible exit in window
        tmin: Worst possible price in window
        p: Profit amount if successful
        l: Loss amount if stopped
        target: Final P&L (can be normalized to pips)
    """
```

### Target Interpretation

| Target Value | Meaning |
|--------------|---------|
| Positive | Signal was profitable - price moved in signal direction |
| Negative | Signal was a loss - stop would have been hit |
| Zero | No FDB signal on this bar |

---

## AO Vector Window Calculation

```python
# Additional feature: Count of AO values from signal to ZLC

pto_vec_fdb_ao_out_s_name = "vaos"      # Sell signal AO vector
pto_vec_fdb_ao_out_b_name = "vaob"      # Buy signal AO vector
pto_vec_fdb_ao_out_s_count = "vaosc"    # Sell signal AO count
pto_vec_fdb_ao_out_b_count = "vaobc"    # Buy signal AO count
pto_vec_fdb_ao_count = "vaoc"           # Combined count

# Calculates: How many bars of AO momentum exist from FDB signal
# until ZLC (zero-line cross) occurs
```

---

## CLI Interface

```python
# jgtml/mxcli.py

def main():
    """
    MX CLI Entry Point.
    
    Arguments:
        -i, --instrument: Instrument symbol (required)
        -t, --timeframe: Timeframe (required)
        --fresh: Force fresh data
        --full/--notfull: Full vs current data
        -rcds: Regenerate CDS before calculation
        -pn, --patternname: Pattern name for TTF
        -sc, --selected-columns: Columns to keep in output
        -ddcc: Don't drop calculated columns
        --mfi: Include MFI indicator
        --gator: Include Gator Oscillator
        --balligator: Include Big Alligator
        --talligator: Include Tide Alligator
        --keepbidask/--rmbidask: Keep/remove bid-ask columns
        -tlidrange: TLID range for data subset (future)
    
    Examples:
        # Default MX calculation
        mxcli -i EUR/USD -t H1
        
        # With pattern name
        mxcli -i EUR/USD -t H1 -pn mz
        
        # Include all Alligators
        mxcli -i SPX500 -t D1 --balligator --talligator
        
        # Force CDS regeneration
        mxcli -i GBPUSD -t H4 -rcds
    """
```

---

## Output Schema

### Full MX DataFrame

```python
{
    # Price data
    "Date": datetime,        # Index
    "Open": float,
    "High": float,
    "Low": float,
    "Close": float,
    "Volume": int,
    
    # Williams indicators
    "ao": float,
    "ac": float,
    "fh": int,               # Fractal High
    "fl": int,               # Fractal Low
    "fdbb": int,             # FDB Buy
    "fdbs": int,             # FDB Sell
    "zlcb": int,             # Zero-line cross up
    "zlcs": int,             # Zero-line cross down
    
    # Alligator
    "jaw": float,
    "teeth": float,
    "lips": float,
    
    # AO Vectors
    "vaos": array,           # Sell AO vector (optional)
    "vaob": array,           # Buy AO vector (optional)
    "vaosc": int,            # Sell AO count
    "vaobc": int,            # Buy AO count
    "vaoc": int,             # Combined count
    
    # TARGET (the key output)
    "target": float,         # P&L in pips or price units
    
    # TTF columns if use_ttf=True
    "zone_sig_H4": str,
    "zone_sig_D1": str,
    ...
}
```

### Selection 1 (_sel)

```python
sel_1_keeping_columns = [
    "High", "Low",
    "fdbs", "fdbb",
    "tmax", "tmin",  # Best/worst prices
    "p", "l",        # Profit/loss amounts
    "target"         # Final P&L
]
```

### Selection 2 (_tnd)

```python
sel_2_keeping_columns = [
    "fdb",           # Signal column
    "target"         # P&L
]
```

---

## File Locations

```python
# MX outputs go to:
$JGTPY_DATA_FULL/targets/mx/

# Example files:
EUR-USD_H1.csv           # Full MX matrix
EUR-USD_H1_sel.csv       # Selected columns
EUR-USD_H1_tnd.csv       # Minimal target-only
```

---

## Configuration

### Default Columns to Keep

```python
ML_DEFAULT_COLUMNS_TO_KEEP = [
    'High', 'Low', 'ao', 'ac',
    'jaw', 'teeth', 'lips',
    'fh', 'fl', 'fdbb', 'fdbs',
    'zlcb', 'zlcs', 'target',
    'vaosc', 'vaobc'
]
```

### Window Configuration

```python
WINDOW_MIN = 1      # Start looking 1 bar after signal
WINDOW_MAX = 150    # Look up to 150 bars forward

# The window determines how far forward we look
# to calculate if the trade would have been profitable
```

---

## Dependencies

```python
import pandas as pd
import numpy as np
import tlid
from jgtpy import JGTPDSP as pds
from jgtutils.jgtos import get_data_path
from mlutils import get_outfile_fullpath
from mlconstants import MX_NS
from jgtutils.jgtconstants import (
    FDBB, FDBS, AO, ZLCB, ZLCS,
    OPEN, LOW, CLOSE, HIGH, DATE,
    FDB_TARGET, VECTOR_AO_FDBS, VECTOR_AO_FDBB,
    ML_DEFAULT_COLUMNS_TO_KEEP
)
```

---

## Service Layer

```python
class MXService:
    """
    MX Service for programmatic access.
    
    Capabilities:
    - Create MX (calculates targets)
    - Read MX (returns cached if exists)
    - Update MX (regenerates with fresh data)
    - Chain from TTF/MLF if needed
    
    Dependencies:
    - MXRequest
    - MLF Service
    """
```

---

## Quality Criteria

✅ **Forward-Looking Target**: Calculates actual P&L over configurable window  
✅ **Signal-Aware**: Only calculates targets for bars with FDB signals  
✅ **Loss Detection**: Identifies when stop would have been hit  
✅ **AO Vector Features**: Counts momentum bars until ZLC  
✅ **TTF Integration**: Includes HTF context when enabled  
✅ **Multiple Outputs**: Full matrix, selection, and minimal versions

---

## Usage in ML Pipeline

```python
# Training a model to predict signal profitability:
import pandas as pd
from sklearn.ensemble import RandomForestClassifier

# Load MX data
mx = pd.read_csv('EUR-USD_H1.csv')

# Create binary target: profitable or not
mx['profitable'] = (mx['target'] > 0).astype(int)

# Feature columns (exclude target and price)
features = ['ao', 'ac', 'jaw', 'teeth', 'lips', 
            'zone_sig_H4', 'zone_sig_D1', 'vaosc', 'vaobc']

X = mx[features]
y = mx['profitable']

# Train classifier
model = RandomForestClassifier()
model.fit(X, y)
```

===============================================
FILE: signal-ordering.spec.md
PATH: jgtml/rispecs/signal-ordering.spec.md
URL:  https://jgtml.jgwill.com/rispecs/signal-ordering.spec.md
===============================================

# Signal Ordering Helper Specification

> Entry Order Generation and Risk Calculation

**Specification Version**: 1.0  
**Module**: `jgtml/SignalOrderingHelper.py`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Executable trading entry orders from FDB signals, with tick-adjusted entry/stop rates, risk calculations, and ready-to-run shell scripts for order placement.

**Achievement Indicator**: Given a signal bar and current bar, produces:
- Tick-adjusted entry and stop rates
- Risk calculation (total, per unit, in pips)
- Executable shell script calling `jgtnewsession`
- Structured order result object

**Value Proposition**: Bridge between raw signal detection and order execution with proper risk management.

---

## Structural Tension

**Current Reality**: FDB signal detected in CDS data with raw high/low prices.

**Desired State**: Complete order specification with spread-adjusted prices, calculated risk, and executable command.

**Natural Progression**: Signal bar → Validate not broken → Validate Alligator → Calculate rates → Calculate risk → Generate script.

---

## Core Functions

### Entry Order Creation

```python
def create_fdb_entry_order(
    i: str,                              # Instrument symbol
    signal_bar: pd.Series,               # Last completed bar with FDB signal
    current_bar: pd.Series,              # Current incomplete bar
    lots: float = 1,                     # Position size
    tick_shift: int = 2,                 # Ticks to add/subtract for spread
    quiet: bool = True,                  # Suppress output
    valid_gator_mouth_open_in_mouth: bool = False,  # Require mouth validation
    validate_signal_out_of_mouth: bool = True,      # Require out-of-mouth
    t: Optional[str] = None,             # Timeframe
    validation_timestamp: Optional[datetime] = None,
    verbose_level: int = 0,
    demo_flag: bool = True               # Use demo broker
) -> tuple[Optional[dict], str]:
    """
    Create entry order from FDB signal.
    
    Returns:
        (order_result, message) - order_result is None if invalid
    
    Order Result Structure:
        {
            "sh": str,           # Executable shell script
            "entry": float,      # Entry rate
            "stop": float,       # Stop rate
            "bs": str,           # "B" or "S"
            "lots": float,       # Position size
            "tlid_id": str,      # Unique ID
            "i": str,            # Instrument
            "t": str,            # Timeframe
            "total_risk": float, # Total monetary risk
            "unit_risk": float,  # Risk per unit
            "pips_risk": float,  # Risk in pips
            "htfsig": str        # HTF context (added later)
        }
    """
```

### Rate Calculation

```python
def get_entry_stop_rate_ticked(
    i: str,                    # Instrument
    bs: str,                   # "B" or "S"
    entry_rate: float,         # Raw entry price
    stop_rate: float,          # Raw stop price
    tick_shift: int = 1,       # Tick adjustment
    rounding_add: int = 2,     # Extra decimal places
    t: Optional[str] = None    # Timeframe
) -> tuple[float, float]:
    """
    Adjust entry and stop rates with tick buffer for spread.
    
    Logic:
        pip_size = get_pips(instrument)  # e.g., 0.0001
        tick_size = pip_size / 10
        
        For Buy:
            entry_rate += tick_size * tick_shift  # Enter higher
            stop_rate -= tick_size * tick_shift   # Stop lower
        
        For Sell:
            entry_rate -= tick_size * tick_shift  # Enter lower
            stop_rate += tick_size * tick_shift   # Stop higher
        
        Round to appropriate decimal places
    
    Returns:
        (adjusted_entry_rate, adjusted_stop_rate)
    """
```

### Risk Calculation

```python
def calculate_entry_risk(
    i: str,                    # Instrument
    bs: str,                   # "B" or "S"
    entry_rate: float,         # Entry price
    stop_rate: float,          # Stop price
    position_size: float,      # Lots/units
    tick_shift: int = 1,
    rounding_add: int = 2,
    t: Optional[str] = None,
    quiet: bool = True,
    verbose_level: int = 0
) -> tuple[float, float, float]:
    """
    Calculate risk metrics for entry order.
    
    Calculation:
        pip_size = get_pips(instrument)
        risk_per_unit = abs(entry_rate - stop_rate)
        total_risk = risk_per_unit * position_size
        risk_in_pips = risk_per_unit / pip_size
    
    Returns:
        (total_risk, risk_per_unit, risk_in_pips)
    """
```

---

## Alligator Validation Functions

### Mouth State Detection

```python
def is_mouth_open(bar: pd.Series, bs: str) -> bool:
    """
    Check if Alligator mouth is open for given direction.
    
    For Buy (bs="B"):
        lips < teeth < jaw
        Alligator lines fan down, mouth opens upward
    
    For Sell (bs="S"):
        lips > teeth > jaw
        Alligator lines fan up, mouth opens downward
    
    Uses columns: 'lips', 'teeth', 'jaw'
    """

def is_big_mouth_open(bar: pd.Series, bs: str) -> bool:
    """
    Check if Big Alligator (34-55-89) mouth is open.
    
    Uses columns: 'blips', 'bteeth', 'bjaw'
    """

def is_bar_out_of_mouth(bar: pd.Series, bs: str) -> bool:
    """
    Check if price bar is outside the Alligator mouth.
    
    For Buy:
        bar.High < bar.Lips (entirely below lips)
        AND mouth not open in opposite direction
    
    For Sell:
        bar.Low > bar.Lips (entirely above lips)
        AND mouth not open in opposite direction
    """

def is_mouth_open_and_bar_out_of_it(bar: pd.Series, bs: str) -> bool:
    """
    Combined check: mouth is open AND bar is out of it.
    
    Returns:
        is_bar_out_of_mouth(bar, bs) AND is_mouth_open(bar, bs)
    """

def valid_gator(
    last_bar_completed: pd.Series,
    current_bar: pd.Series,
    bs: str
) -> bool:
    """
    Validate Alligator state for both signal bar and current bar.
    
    Both bars must have:
        - Mouth open in signal direction
        - Price out of mouth
    
    This ensures the Alligator "agrees" with the trade direction.
    """
```

### Signal Crossing Detection

```python
def is_fdbsignal_crossed_t(bar: pd.Series, bs: str, tcol: str) -> bool:
    """
    Check if FDB signal bar crosses a specific Alligator line.
    
    For Buy: bar.High < bar[tcol] (price below the line)
    For Sell: bar.Low > bar[tcol] (price above the line)
    """

def is_fdbsignal_in_big_mouth(bar: pd.Series, bs: str) -> bool:
    """Check if signal is within Big Alligator lips."""
    
def is_fdbsignal_in_big_mouth_teeth(bar: pd.Series, bs: str) -> bool:
    """Check if signal is within Big Alligator teeth."""
```

---

## Script Generation

```python
def generate_entry_order_script(
    lots: float,
    entry_rate: float,
    stop_rate: float,
    instrument: str,
    buysell: str,
    tlid_id: Optional[str] = None,
    t: Optional[str] = None,
    validation_timestamp_str: str = "",
    demo_flag: bool = True,
    total_risk: Optional[float] = None,
    risk_per_unit: Optional[float] = None,
    risk_in_pips: Optional[float] = None,
    extra_scripting_output: Optional[str] = None
) -> str:
    """
    Generate executable shell script for jgtnewsession.
    
    Output Format:
    ```sh
    ### --- COPY FROM HERE ---
    demo_arg="--demo"  # or --real
    # FDB Buy Entry EUR/USD H4 - bts/now:2026-01-31 14:00/2026-01-31 14:05
    risk_in_pips=25.5
    instrument="EUR/USD";timeframe="H4";bs="B"
    tlid_id=260131140500;lots=1
    entry_rate=1.0950;stop_rate=1.0900
    jgtnewsession $tlid_id $instrument $timeframe $entry_rate $stop_rate $bs $lots $demo_arg
    zone=B-B-S-N-N  # extra_scripting_output
    ### ---- COPY TO HERE ---
    ```
    """
```

---

## Order Result Object

```python
def build_order_result_object(
    lots: float,
    entry_rate: float,
    stop_rate: float,
    buysell: str,
    tlid_id: str,
    output_script: str,
    i: str,
    t: str,
    total_risk: Optional[float] = None,
    risk_per_unit: Optional[float] = None,
    risk_in_pips: Optional[float] = None
) -> dict:
    """
    Build structured order result object.
    
    Returns:
        {
            "sh": output_script,
            "entry": entry_rate,
            "stop": stop_rate,
            "bs": buysell,
            "lots": lots,
            "tlid_id": tlid_id,
            "i": i,
            "t": t,
            "total_risk": total_risk,
            "unit_risk": risk_per_unit,
            "pips_risk": risk_in_pips
        }
    """
```

---

## Validation Flow

```
1. Check FDB Signal
   └── signal_bar['fdb'] == 1 (buy) or -1 (sell)
   └── Return None if no signal

2. Determine Entry/Stop Rates
   └── Buy: entry=AskHigh, stop=BidLow
   └── Sell: entry=BidLow, stop=AskHigh
   └── Apply tick_shift adjustment

3. Check Signal Not Broken
   └── Buy: current.high <= entry AND current.low >= stop
   └── Sell: current.low >= entry AND current.high <= stop
   └── Return None if broken

4. Validate Alligator (optional)
   └── valid_gator_mouth_open_in_mouth: full mouth validation
   └── validate_signal_out_of_mouth: bar outside lips
   └── Return None if validation fails

5. Calculate Risk
   └── total_risk, unit_risk, pips_risk

6. Generate Script
   └── jgtnewsession command with all parameters

7. Build Result Object
   └── Complete order specification
```

---

## Column Dependencies

```python
# From CDS data
HIGH = "High"
LOW = "Low"
FDB = "fdb"           # -1 (sell), 0 (none), 1 (buy)
ASKHIGH = "AskHigh"   # Ask price high
ASKLOW = "AskLow"     # Ask price low
BIDHIGH = "BidHigh"   # Bid price high
BIDLOW = "BidLow"     # Bid price low

# Regular Alligator
JAW = "jaw"           # 13-period SMMA, shifted 8
TEETH = "teeth"       # 8-period SMMA, shifted 5
LIPS = "lips"         # 5-period SMMA, shifted 3

# Big Alligator
BJAW = "bjaw"         # 89-period SMMA
BTEETH = "bteeth"     # 55-period SMMA
BLIPS = "blips"       # 34-period SMMA
```

---

## Dependencies

```python
import datetime
from typing import Optional
import pandas as pd
import tlid                    # Timestamp ID generation
from jgtutils import iprops    # Instrument properties
from jgtutils.jgtconstants import (
    HIGH, LOW, FDB, ASKHIGH, ASKLOW, BIDHIGH, BIDLOW,
    JAW, TEETH, LIPS, BJAW, BTEETH, BLIPS, DATE
)
```

---

## Quality Criteria

✅ **Tick Adjustment**: Entry/stop adjusted for spread  
✅ **Multi-layer Validation**: Signal → Broken → Alligator  
✅ **Risk Calculation**: Total, per-unit, and pips  
✅ **Executable Output**: Ready-to-run shell script  
✅ **Structured Result**: Complete order object  
✅ **Demo/Real Support**: Flag for broker mode

===============================================
FILE: ttf.spec.md
PATH: jgtml/rispecs/ttf.spec.md
URL:  https://jgtml.jgwill.com/rispecs/ttf.spec.md
===============================================

# TTF (Cross-Timeframe Features) Specification

> Higher Timeframe Context for Signal Validation

**Specification Version**: 1.0  
**Modules**: `jgtml/ttfcli.py`, `jgtml/ptottf.py`, `jgtml/ttfsvc.py`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Cross-timeframe feature datasets that enrich current timeframe signals with context from all higher timeframes (H1 signal knows D1/W1/MN conditions).

**Achievement Indicator**: Running `ttfcli -i EUR/USD -t H1 -pn mz` produces:
- CSV file with base H1 CDS data
- Additional columns for each HTF: `zone_sig_H4`, `zone_sig_D1`, etc.
- Pattern name metadata stored for downstream MLF consumption

**Value Proposition**: Never trade H1 signals against D1 trend - TTF makes HTF context instantly available in every row.

---

## Structural Tension

**Current Reality**: CDS data exists for each timeframe independently.

**Desired State**: Single dataset with current TF features plus all HTF features aligned by timestamp.

**Natural Progression**: CDS → Get HTF datasets → Merge by timestamp → Save TTF CSV.

---

## Data Pipeline Position

```
PDS → IDS → CDS → [TTF] → MLF → MX
                    ↑
              Current stage
```

**Dependencies**: Requires CDS data for instrument across all timeframes.

---

## Core Function: create_ttf_csv

```python
def create_ttf_csv(
    i: str,                                  # Instrument symbol
    t: str,                                  # Target timeframe
    use_full: bool = False,                  # Full vs current data
    use_fresh: bool = True,                  # Force refresh CDS first
    quotescount: int = -1,                   # Bars to fetch (-1 = default)
    force_read: bool = False,                # Force read existing CDS
    dropna: bool = True,                     # Drop NaN rows
    quiet: bool = True,                      # Suppress output
    columns_list_from_higher_tf: List[str] = None,  # Columns to get from HTF
    not_needed_columns: List[str] = None,    # Columns to remove
    dropna_volume: bool = True,              # Drop zero-volume rows
    pn: str = "ttf",                         # Pattern name
    also_output_sel_csv: bool = False,       # Also save selected columns
    args: argparse.Namespace = None          # CLI args for settings
) -> pd.DataFrame:
    """
    Create TTF dataset with cross-timeframe features.
    
    Algorithm:
        1. Load settings for pattern (pn) to get columns_list_from_higher_tf
        2. Get higher timeframe array: H1 -> [H4, D1, W1, MN]
        3. If use_fresh: refresh all dependent CDS data
        4. Load CDS workset (dict of timeframe -> DataFrame)
        5. Get base DataFrame for target timeframe
        6. For each HTF in workset:
           - For each column in columns_list_from_higher_tf:
             - Create new column: {col}_{htf} (e.g., zone_sig_D1)
             - For each row in base DF:
               - Find latest HTF row <= current timestamp
               - Copy HTF column value to base DF
        7. Write pattern column metadata
        8. Drop not_needed_columns
        9. Save to CSV
    
    Returns:
        DataFrame with merged TTF features
    
    Output Path:
        $JGTPY_DATA/ttf/{instrument}_{timeframe}_{patternname}.csv
    """
```

---

## Timeframe Hierarchy

```python
def get_higher_tf_array(t: str) -> List[str]:
    """
    Get array of all timeframes higher than target.
    
    Examples:
        "m1" -> ["m5", "m15", "m30", "H1", "H4", "D1", "W1", "MN"]
        "H1" -> ["H4", "D1", "W1", "MN"]
        "D1" -> ["W1", "MN"]
        "W1" -> ["MN"]
        "MN" -> []
    """
```

Timeframe Order (low to high):
```
m1 < m5 < m15 < m30 < H1 < H2 < H3 < H4 < H6 < H8 < D1 < W1 < MN
```

---

## Pattern Configuration

### Pattern Columns Definition

Patterns are defined by which columns to copy from higher timeframes.

```python
# Pattern: mz (MFI + Zone)
COLUMNS_MZ = ["mfi_sig", "zone_sig", "ao"]

# Pattern: full (all Williams indicators)  
COLUMNS_FULL = ["mfi_sig", "zone_sig", "ao", "ac", "fdb", "fh", "fl"]
```

### Pattern Metadata Storage

```python
def write_patternname_columns_list(
    i: str,
    t: str,
    use_full: bool,
    columns_list: List[str],
    pn: str = "ttf"
) -> None:
    """
    Save pattern column metadata for MLF consumption.
    
    Output Path:
        $JGTPY_DATA/ttf/{instrument}_{timeframe}_{pn}_columns.json
    
    Format:
        ["zone_sig", "mfi_sig", "ao", "zone_sig_H4", "zone_sig_D1", ...]
    """
```

---

## CLI Interface

```python
# jgtml/ttfcli.py

def main():
    """
    TTF CLI Entry Point.
    
    Arguments:
        -i, --instrument: Instrument symbol (required)
        -t, --timeframe: Target timeframe (required)
        -pn, --patternname: Pattern name (default: "ttf")
        --fresh: Force refresh dependent CDS data
        --full: Use full historical data
        -n, --quotescount: Number of bars to include
        -clh, --columns_list_from_higher_tf: Custom column list
    
    Examples:
        # Default TTF with standard columns
        ttfcli -i EUR/USD -t H1
        
        # MZ pattern (MFI + Zone)
        ttfcli -i EUR/USD -t H1 -pn mz
        
        # Custom column selection
        ttfcli -i SPX500 -t D1 -clh ao ac zone_sig
        
        # Force refresh CDS before TTF
        ttfcli -i GBPUSD -t H4 --fresh
    """
```

---

## Programmatic API

```python
def generate_ttf_for_pattern(
    instrument: str,
    timeframe: str,
    patternname: str = "ttf",
    use_full: bool = True,
    use_fresh: bool = False,
    quotescount: int = -1,
    columns_list_from_higher_tf: List[str] = None,
    args: argparse.Namespace = None
) -> pd.DataFrame:
    """
    Programmatic helper mirroring CLI behavior for JTC integrations.
    
    Includes JGTTracer integration for observability.
    """
```

---

## Output Schema

### Column Naming Convention

```
{original_column}_{higher_timeframe}

Examples:
- zone_sig_H4    # Zone signal from H4
- zone_sig_D1    # Zone signal from D1
- mfi_sig_W1     # MFI signal from W1
- ao_H4          # AO from H4
```

### Output DataFrame Structure

```python
# For H1 with mz pattern, output columns:
{
    # Base CDS columns
    "Date": datetime,      # Index
    "Open": float,
    "High": float,
    "Low": float,
    "Close": float,
    "Volume": int,
    # Williams indicators
    "ao": float,
    "zone_sig": str,       # "B", "S", "N"
    "mfi_sig": str,
    "fdb": int,
    # Regular Alligator
    "jaw": float,
    "teeth": float,
    "lips": float,
    # HTF Context (added by TTF)
    "zone_sig_H4": str,
    "zone_sig_D1": str,
    "zone_sig_W1": str,
    "zone_sig_MN": str,
    "mfi_sig_H4": str,
    "mfi_sig_D1": str,
    "mfi_sig_W1": str,
    "mfi_sig_MN": str,
    "ao_H4": float,
    "ao_D1": float,
    "ao_W1": float,
    "ao_MN": float,
}
```

---

## Merge Algorithm

### Timestamp Alignment

```python
# For each row in base DataFrame:
for ii in df.index:
    date = ii  # Timestamp of current row
    
    # For each higher timeframe dataset:
    for key_tf, v in workset.items():
        if key_tf != t:  # Skip current timeframe
            v_sorted = v.sort_index()
            
            # Find latest HTF data <= current timestamp
            data = v_sorted[v_sorted.index <= date]
            
            if not data.empty:
                latest_data = data.iloc[-1]  # Most recent HTF bar
                
                for col in columns_list_from_higher_tf:
                    new_col_name = f"{col}_{key_tf}"
                    df.at[ii, new_col_name] = latest_data[col]
```

### Why This Works

- H1 bar at 14:00 gets D1 context from the last completed D1 bar (yesterday's close)
- W1 context comes from the most recent completed weekly bar
- This ensures we never have look-ahead bias (only using data that was available at the time)

---

## File Locations

```python
def get_ttf_outfile_fullpath(
    i: str,
    t: str,
    use_full: bool,
    suffix: str = "",
    pn: str = "ttf"
) -> str:
    """
    Get output file path for TTF CSV.
    
    Path Construction:
        base_dir = $JGTPY_DATA/ttf (current) or $JGTPY_DATA_FULL/ttf (full)
        filename = {instrument}_{timeframe}_{patternname}{suffix}.csv
        
    Examples:
        JGTPY_DATA=/src/jgtml/data/current
        -> /src/jgtml/data/current/ttf/EUR-USD_H1_mz.csv
        -> /src/jgtml/data/current/ttf/EUR-USD_H1_mz_columns.json
    """
```

---

## Configuration via Settings

```python
def get_settings() -> dict:
    """
    Load TTF settings from ~/.jgt/config.json or pattern-specific config.
    
    Settings Keys:
        ttf_columns_to_remove: List[str]  # Columns to drop from output
        columns_to_remove: List[str]      # Additional columns to remove
    """
```

---

## Dependencies

```python
import pandas as pd
from jgtpy import JGTCDSSvc as svc         # CDS service for data loading
from jgtutils import jgtpov as jpov        # Timeframe ordering
from mlutils import drop_columns_if_exists, dropna_volume_in_dataframe
from mlconstants import TTF_NOT_NEEDED_COLUMNS_LIST
from mldatahelper import (
    get_settings,
    get_ttf_outfile_fullpath,
    write_patternname_columns_list,
    pndata__read_new_pattern_columns_list
)
```

---

## Quality Criteria

✅ **HTF Context**: Every row includes data from all higher timeframes  
✅ **No Look-Ahead**: Only uses data available at each timestamp  
✅ **Pattern Support**: Custom column selection via pattern names  
✅ **Metadata Storage**: Column definitions saved for MLF consumption  
✅ **Fresh Data Option**: Can force-refresh dependent CDS data  
✅ **Tracing**: JGTTracer integration for observability

---

## Downstream Usage

```python
# TTF is consumed by MLF to add lagging features:
from jgtml.mlfcli import generate_mlf_for_pattern

df = generate_mlf_for_pattern(
    "EUR/USD", "H1",
    patternname="mz",           # Uses TTF output
    lag_period=1,               # 1-bar lag
    total_lagging_periods=5     # 5 lagged versions
)
```

===============================================
FILE: app.specs.md
PATH: jgtpy/rispecs/app.specs.md
URL:  https://jgtpy.jgwill.com/rispecs/app.specs.md
===============================================

# JGTpy Application Specification

> Master specification for the JGT Data Services Package

**Specification Version**: 1.0  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: A complete data transformation pipeline that converts raw price data into signal-rich trading intelligence, enriched with Williams 5 Dimensions indicators and ready for analysis.

**Achievement Indicator**: Users can run `cdscli --fresh` and obtain CDS files containing all Williams indicators, trading signals (FDB, ZLC, Saucer, Zone), and chart-ready data.

**Value Proposition**: Transform scattered price data into actionable trading signals through a systematic pipeline: PDS → IDS → CDS → ADS.

---

## Application Overview

JGTpy is a Python package that:
1. Wraps jgtfxcon for price data acquisition (PDS)
2. Enriches data with Williams indicators via jgtapy (IDS)
3. Generates trading signals for entries/exits (CDS)
4. Produces charts and visualizations (ADS)
5. Runs as automated service with scheduling (jgtservice)

---

## Structural Tension

**Current Reality**: Raw OHLCV price data lacks the derived indicators and signals needed for Williams-based trading decisions.

**Desired State**: Complete signal-rich datasets ready for trading analysis, served through CLI, API, or automated daemon.

**Natural Progression**: Each layer builds upon the previous, transforming raw data into actionable intelligence: PDS → IDS → CDS → ADS.

---

## Data Pipeline Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    JGTpy Data Pipeline                       │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────┐   │
│  │   PDS   │ →  │   IDS   │ →  │   CDS   │ →  │   ADS   │   │
│  │ (Price) │    │(Indicators)│  │(Signals) │   │(Charts) │   │
│  └─────────┘    └─────────┘    └─────────┘    └─────────┘   │
│       ↑              ↑              ↑              ↑        │
│   jgtfxcon       jgtapy         JGTCDS.py      JGTADS.py   │
│                                                              │
└─────────────────────────────────────────────────────────────┘
```

### Layer Descriptions

| Layer | Module | Description | Key Columns Added |
|-------|--------|-------------|-------------------|
| PDS | JGTPDSRequest | Raw OHLCV from broker | Date, Open, High, Low, Close, Volume |
| IDS | JGTIDS | Williams indicators | jaw, teeth, lips, ao, ac, fh, fl, mfi |
| CDS | JGTCDS | Trading signals | fdb, fdbb, fdbs, zlcB, zlcS, bz, sz |
| ADS | JGTADS | Visualization data | Chart-ready format |

---

## Core Signals (CDS)

### Signal Reference

| Signal | Column | Description |
|--------|--------|-------------|
| FDB | fdb | Fractal Divergent Bar (buy=-1, sell=1) |
| FDBB | fdbb | FDB Buy signal |
| FDBS | fdbs | FDB Sell signal |
| ZLC | zlc | Zero Line Crossing |
| ZLCB | zlcB | ZLC Buy signal |
| ZLCS | zlcS | ZLC Sell signal |
| BZ | bz | Buy Zone signal |
| SZ | sz | Sell Zone signal |
| SB | sb | Saucer Buy signal |
| SS | ss | Saucer Sell signal |
| FS | fs | Fractal Sell |
| FB | fb | Fractal Buy |

### FDB Signal Logic

**Fractal Divergent Bar (FDB)** is the primary entry signal:

```python
# Buy FDB conditions:
# 1. Price makes new fractal low (fl set)
# 2. AO does NOT make new low (bullish divergence)
# 3. Bar close above teeth line (Alligator confirmation)

# Sell FDB conditions:
# 1. Price makes new fractal high (fh set)
# 2. AO does NOT make new high (bearish divergence)
# 3. Bar close below teeth line
```

---

## CLI Tools

### jgtcli - Price/Indicator Data

```bash
# Fetch fresh price data with indicators
jgtcli -i EUR/USD -t H4 --fresh

# Multiple instruments and timeframes
jgtcli -i "EUR/USD,SPX500" -t "H1,H4,D1" --fresh

# With specific bar count
jgtcli -i EUR/USD -t H4 -c 500
```

### cdscli - Signal Generation

```bash
# Generate fresh CDS (signals)
cdscli -i EUR/USD -t H4 --fresh

# All configured instruments/timeframes
cdscli --all --fresh

# Use cached IDS data
cdscli -i EUR/USD -t H4
```

### jgtads - Advanced Charts

```bash
# Display chart
jgtads -i EUR/USD -t H4 --show

# Save chart to file
jgtads -i EUR/USD -t H4 --save_figure charts/

# Auto-named by timeframe
jgtads -i EUR/USD -t H4 --save_figure charts/ --save_figure_as_timeframe
```

### jgtservice - Daemon Mode

```bash
# One-time refresh
jgtservice --refresh-once -i EUR/USD -t H1

# Continuous daemon
jgtservice --daemon --all

# Web API server
jgtservice --web --port 8080

# Check status
jgtservice --status
```

---

## Type Definitions

```python
from typing import Dict, Any, Optional, List
import pandas as pd
from dataclasses import dataclass

# Instrument format: "EUR/USD", "SPX500"
Instrument = str

# Timeframe format: "m1", "m5", "m15", "m30", "H1", "H4", "D1", "W1", "M1"
Timeframe = str

@dataclass
class CDSData:
    """Chaos Data Service record with signals"""
    instrument: Instrument
    timeframe: Timeframe
    date: str
    open: float
    high: float
    low: float
    close: float
    volume: float
    # Alligator
    jaw: float
    teeth: float
    lips: float
    # Oscillators
    ao: float
    ac: float
    # Fractals
    fh: Optional[float]  # Fractal high
    fl: Optional[float]  # Fractal low
    # Signals
    fdb: int        # -1=buy, 0=none, 1=sell
    fdbb: bool      # FDB buy
    fdbs: bool      # FDB sell
    zlcB: bool      # ZLC buy
    zlcS: bool      # ZLC sell
    bz: bool        # Buy zone
    sz: bool        # Sell zone
    # MFI
    mfi: float
    mfi_signal: str  # "green", "squat", "fade", "fake"

def create_cds(
    instrument: Instrument,
    timeframe: Timeframe,
    bars: int = 335
) -> pd.DataFrame: ...

def read_cds(
    instrument: Instrument,
    timeframe: Timeframe
) -> pd.DataFrame: ...

def refresh_cds(
    instruments: List[Instrument],
    timeframes: List[Timeframe],
    parallel: bool = True
) -> Dict[str, Any]: ...
```

---

## Creative Advancement Scenarios

### Scenario: Fresh Signal Generation

**Desired Outcome**: Up-to-date CDS with all Williams signals

**Current Reality**: User starts trading session

**Natural Progression**:
1. User runs: `cdscli -i EUR/USD -t H4 --fresh`
2. JGTCDS invokes JGTIDS for indicator calculation
3. JGTIDS calls jgtapy for Alligator, AO, AC, Fractals, MFI
4. JGTCDS applies signal detection logic
5. CDS file written: `$JGTPY_DATA/cds/EUR-USD_H4.csv`

**Resolution**: CDS contains all indicators + signals, ready for analysis

### Scenario: Automated Refresh Service

**Desired Outcome**: Data stays current without manual intervention

**Current Reality**: Need continuous updates during trading hours

**Natural Progression**:
1. Start daemon: `jgtservice --daemon --all`
2. Scheduler monitors timeframe completion
3. On H1 close (e.g., 14:00), triggers refresh
4. PDS→IDS→CDS pipeline runs for all instruments
5. Optional cloud sync to Dropbox

**Resolution**: All data files stay current automatically

### Scenario: Terminal Chart Display

**Desired Outcome**: Visual analysis of Williams patterns

**Current Reality**: Want to see Alligator and signals on chart

**Natural Progression**:
1. User runs: `jgtads -i EUR/USD -t H4 --show`
2. JGTADS loads CDS data
3. mplfinance renders candlestick chart
4. Alligator lines overlaid (Jaw, Teeth, Lips)
5. Signals marked on chart (FDB, ZLC, etc.)

**Resolution**: Interactive chart displays complete Williams analysis

---

## File Storage

```
$JGTPY_DATA/
├── pds/                 # Price Data Service
│   ├── EUR-USD_H4.csv
│   └── ...
├── ids/                 # Indicator Data Service (optional cache)
│   └── ...
├── cds/                 # Chaos Data Service (primary)
│   ├── EUR-USD_H4.csv
│   └── ...
└── ads/                 # Advanced Data (charts/exports)
    └── ...
```

---

## Module Structure

```
jgtpy/
├── __init__.py           # Package exports
├── jgtcli.py             # jgtcli entry point
├── cdscli.py             # cdscli entry point
├── JGTIDS.py             # Indicator Data Service
├── JGTIDSRequest.py      # IDS request handling
├── JGTIDSSvc.py          # IDS service layer
├── JGTCDS.py             # Chaos Data Service
├── JGTCDSRequest.py      # CDS request handling
├── JGTCDSSvc.py          # CDS service layer
├── JGTADS.py             # Advanced Data Service
├── JGTADSRequest.py      # ADS request handling
├── JGTPDHelper.py        # PD utilities
├── JGTPDSP.py            # PDS processing
├── JGTPDSRequest.py      # PDS request handling
├── jgtservice.py         # Daemon service
├── jgtapycli.py          # Indicator CLI
├── jgtapyhelper.py       # jgtapy integration
├── glyph_cli.py          # Terminal glyphs
├── alligator_mouth_water.py  # Alligator state analysis
└── service/              # Service components
```

---

## Integration with JGT Ecosystem

```
jgtcore (configuration)
    ↓
jgtutils (utilities)
    ↓
jgtapy (indicators)
    ↓
jgtfxcon (broker connection)
    ↓
jgtpy (this package) ← Central data layer
    ↓ provides CDS data
jgtml (ML/analysis)
    ↓
jgt-data-server (REST API)
    ↓
jgt-code (terminal agent)
```

---

## Quality Criteria

✅ **Complete Pipeline**: PDS→IDS→CDS→ADS fully implemented  
✅ **Williams Native**: All 5 Dimensions as first-class data  
✅ **CLI Complete**: Every operation available via command line  
✅ **Daemon Mode**: Automated refresh with scheduling  
✅ **Parallel Processing**: Multi-instrument concurrent refresh  
✅ **API Ready**: RESTful endpoints via jgtservice

===============================================
FILE: cds.spec.md
PATH: jgtpy/rispecs/cds.spec.md
URL:  https://jgtpy.jgwill.com/rispecs/cds.spec.md
===============================================

# CDS (Chaos Data Service) Specification

> FDB Signal Generation and Williams Indicator Processing

**Specification Version**: 1.0  
**Modules**: `jgtpy/JGTCDS.py`, `jgtpy/JGTCDSSvc.py`, `jgtpy/cdscli.py`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: CDS datasets containing price data enriched with complete Williams Trading System signals (FDB, ZLC, Fractals, Zone, MFI) ready for trading decisions.

**Achievement Indicator**: Running `cdscli -i EUR/USD -t H1` produces:
- CSV file with OHLCV + all Williams indicators
- FDB signals (fdbb=1, fdbs=1) marking entry opportunities
- Zone coloring (B=buying, S=selling, N=neutral)
- All three Alligator types (regular, big, tide)

**Value Proposition**: Transform raw price data into actionable Williams signals in one command.

---

## Structural Tension

**Current Reality**: PDS files with raw OHLCV price data.

**Desired State**: CDS files with complete Williams indicator suite and FDB signals.

**Natural Progression**: PDS → Add Alligator → Add AO/AC → Add Fractals → Add FDB → Add Zone → Save CDS.

---

## Data Pipeline Position

```
PDS → IDS → [CDS] → TTF → MLF → MX
             ↑
       Current stage
```

**Dependencies**: Requires PDS (price data) from jgtfxcon or cached files.

---

## Core Function: createFromPDSFileToCDSFile

```python
def createFromPDSFileToCDSFile(
    instrument: str,
    timeframe: str,
    columns_to_remove: List[str] = None,
    quiet: bool = True,
    tlid_range: str = None,
    use_full: bool = False,
    rq: JGTCDSRequest = None,
    use_fresh: bool = True,
    keep_bid_ask: bool = True,
    gator_oscillator_flag: bool = False,
    mfi_flag: bool = True,
    balligator_flag: bool = False,
    balligator_period_jaws: int = 89,
    largest_fractal_period: int = 89,
    talligator_flag: bool = False,
    talligator_period_jaws: int = 377,
    mouth_water_flag: bool = False,
    viewpath: bool = False,
    quotescount: int = 300,
    dropna_volume: bool = True
) -> Tuple[str, pd.DataFrame]:
    """
    Create CDS file from PDS source.
    
    Algorithm:
        1. Create JGTCDSRequest from parameters
        2. Load PDS data (fresh or cached)
        3. Apply indicators via JGTIDS
        4. Calculate FDB signals
        5. Calculate Zone coloring
        6. Write CDS to file
        7. Update zone cache
    
    Returns:
        (filepath, dataframe) - path to CDS file and the DataFrame
    
    Output Path:
        $JGTPY_DATA/cds/{instrument}_{timeframe}.csv
    """
```

---

## CDS Request Object

```python
@dataclass
class JGTCDSRequest:
    """Request configuration for CDS generation."""
    instrument: str = ""
    timeframe: str = ""
    quotescount: int = 300
    use_full: bool = False
    use_fresh: bool = True
    keep_bid_ask: bool = True
    dropna_volume: bool = True
    
    # Indicator flags
    gator_oscillator_flag: bool = False
    mfi_flag: bool = True
    balligator_flag: bool = False
    talligator_flag: bool = False
    mouth_water_flag: bool = False
    
    # Alligator periods
    balligator_period_jaws: int = 89
    talligator_period_jaws: int = 377
    largest_fractal_period: int = 89
    
    viewpath: bool = False
    
    def talligator_fix_quotescount(self):
        """Ensure enough bars for Tide Alligator calculation."""
        if self.talligator_flag:
            min_bars = self.talligator_period_jaws * 2
            if self.quotescount < min_bars:
                self.quotescount = min_bars
```

---

## CLI Interface

```python
# jgtpy/cdscli.py

def main():
    """
    CDS CLI Entry Point.
    
    Arguments:
        -i, --instrument: Instrument symbol (required)
        -t, --timeframe: Timeframe (required)
        --fresh: Force fresh data fetch
        --full: Use full historical data
        -n, --quotescount: Number of bars
        --mfi: Include MFI indicator (default: True)
        --gator: Include Gator Oscillator
        --balligator: Include Big Alligator (89)
        --talligator: Include Tide Alligator (377)
        --mouth-water: Include Mouth Water analysis
        --largest-fractal-period: Fractal lookback period
        --viewpath: Show output path only
        --dropna-volume: Drop zero-volume bars
        --ads: Show chart after generation
    
    Examples:
        # Default CDS generation
        cdscli -i EUR/USD -t H1
        
        # Multiple instruments/timeframes
        cdscli -i EUR/USD,GBP/USD -t H1,H4
        
        # Include all Alligators
        cdscli -i SPX500 -t D1 --balligator --talligator
        
        # Fresh data with chart display
        cdscli -i GBPUSD -t H4 --fresh --ads
    """
```

---

## Output Schema

### Complete CDS DataFrame

```python
{
    # Price Data
    "Date": datetime,        # Index
    "Open": float,
    "High": float,
    "Low": float,
    "Close": float,
    "Volume": int,
    
    # Bid/Ask (optional)
    "BidOpen": float,
    "BidHigh": float,
    "BidLow": float,
    "BidClose": float,
    "AskOpen": float,
    "AskHigh": float,
    "AskLow": float,
    "AskClose": float,
    
    # Regular Alligator (5-8-13)
    "jaw": float,            # 13-period SMMA, shift 8
    "teeth": float,          # 8-period SMMA, shift 5
    "lips": float,           # 5-period SMMA, shift 3
    
    # Big Alligator (34-55-89) - if enabled
    "bjaw": float,
    "bteeth": float,
    "blips": float,
    
    # Tide Alligator (144-233-377) - if enabled
    "tjaw": float,
    "tteeth": float,
    "tlips": float,
    
    # Awesome Oscillator
    "ao": float,             # 5-34 SMA difference on medians
    "aocolor": str,          # "green" or "red"
    
    # Accelerator Oscillator
    "ac": float,             # AO - 5-period SMA of AO
    "accolor": str,          # "green" or "red"
    
    # Fractals
    "fh": int,               # Fractal High (1 or 0)
    "fl": int,               # Fractal Low (1 or 0)
    "fh3": int,              # 3-bar fractal high
    "fl3": int,              # 3-bar fractal low
    "fh5": int,              # 5-bar fractal high
    "fl5": int,              # 5-bar fractal low
    
    # Divergent Bars (FDB signals)
    "fdb": int,              # -1 (sell), 0 (none), 1 (buy)
    "fdbb": int,             # FDB Buy (1 or 0)
    "fdbs": int,             # FDB Sell (1 or 0)
    
    # Zero Line Cross
    "zlc": int,              # Zero line cross direction
    "zlcb": int,             # ZLC Buy
    "zlcs": int,             # ZLC Sell
    
    # Zone
    "bz": int,               # Buying zone (1 or 0)
    "sz": int,               # Selling zone (1 or 0)
    "zcol": str,             # Zone color: "B", "S", "N"
    "zone_sig": str,         # Zone signal
    
    # MFI (if enabled)
    "mfi": float,            # Market Facilitation Index
    "mfi_sig": str,          # MFI signal type
    
    # Bar characteristics
    "bar_height": float,     # High - Low
}
```

---

## Service Layer

```python
class JGTCDSSvc:
    """CDS Service for programmatic access."""
    
    @staticmethod
    def create(rq: JGTCDSRequest) -> pd.DataFrame:
        """Create CDS from request object."""
    
    @staticmethod
    def get(
        instrument: str,
        timeframe: str,
        use_full: bool = False,
        use_fresh: bool = True,
        quotescount: int = -1,
        quiet: bool = True,
        force_read: bool = False
    ) -> pd.DataFrame:
        """Get CDS, creating if needed."""
    
    @staticmethod
    def get_higher_cdf_datasets(
        instrument: str,
        timeframe: str,
        use_full: bool = False,
        use_fresh: bool = False,
        quotescount: int = -1,
        quiet: bool = True,
        force_read: bool = False
    ) -> Dict[str, pd.DataFrame]:
        """
        Get CDS for all timeframes higher than specified.
        Used by TTF for cross-timeframe analysis.
        
        Returns:
            {"H1": df, "H4": df, "D1": df, "W1": df, "MN": df}
        """
    
    @staticmethod
    def zone_update_from_cdf(
        instrument: str,
        timeframe: str,
        cdf: pd.DataFrame,
        quiet: bool = True
    ) -> Tuple[str, dict]:
        """Update zone cache from CDS DataFrame."""
```

---

## Indicator Calculation (JGTIDS)

```python
def ids_add_indicators(
    dfsrc: pd.DataFrame,
    dropnavalue: bool = True,
    quiet: bool = True,
    cleanupOriginalColumn: bool = True,
    cc: JGTChartConfig = None,
    rq: JGTIDSRequest = None
) -> pd.DataFrame:
    """
    Add Williams Trading System indicators to DataFrame.
    
    Calculation Order:
        1. Alligator (jaw, teeth, lips)
        2. Big Alligator (bjaw, bteeth, blips) - if enabled
        3. Tide Alligator (tjaw, tteeth, tlips) - if enabled
        4. Awesome Oscillator (ao, aocolor)
        5. Accelerator Oscillator (ac, accolor)
        6. Gator Oscillator - if enabled
        7. Fractals (fh, fl, fh3, fl3, fh5, fl5)
        8. FDB signals (fdb, fdbb, fdbs)
        9. Zero Line Cross (zlc, zlcb, zlcs)
        10. Zone (bz, sz, zcol)
        11. MFI - if enabled
        12. AO/Price peaks - if enabled
    
    Returns:
        DataFrame with all indicators added
    """
```

---

## Williams Five Dimensions

### 1. Fractal (Space)
```python
# Fractal High: High[i-2] < High[i] > High[i+2] for 5-bar
# Fractal Low: Low[i-2] > Low[i] < Low[i+2] for 5-bar
```

### 2. Momentum (AO)
```python
# AO = SMA(median, 5) - SMA(median, 34)
# median = (High + Low) / 2
# aocolor = "green" if AO > AO[-1] else "red"
```

### 3. Acceleration (AC)
```python
# AC = AO - SMA(AO, 5)
# accolor = "green" if AC > AC[-1] else "red"
```

### 4. Zone
```python
# Buying Zone: 2+ consecutive green AO AND 2+ consecutive green AC
# Selling Zone: 2+ consecutive red AO AND 2+ consecutive red AC
# Neutral: Neither condition met
```

### 5. Balance Line (Alligator)
```python
# Jaw: SMMA(median, 13), shifted 8 bars
# Teeth: SMMA(median, 8), shifted 5 bars
# Lips: SMMA(median, 5), shifted 3 bars
```

---

## FDB Signal Logic

```python
def calculate_fdb(df: pd.DataFrame) -> pd.DataFrame:
    """
    Calculate Fractal Divergent Bar signals.
    
    FDB Buy (fdbb=1):
        - Fractal Low present (fl=1)
        - Close > Teeth
        - Bar forms below Alligator teeth
        - AO conditions met (green preferred)
    
    FDB Sell (fdbs=1):
        - Fractal High present (fh=1)
        - Close < Teeth
        - Bar forms above Alligator teeth
        - AO conditions met (red preferred)
    
    fdb = fdbb - fdbs  # Combined: 1 (buy), -1 (sell), 0 (none)
    """
```

---

## File Locations

```python
# CDS output paths:
$JGTPY_DATA/cds/{instrument}_{timeframe}.csv       # Current
$JGTPY_DATA_FULL/cds/{instrument}_{timeframe}.csv  # Full historical

# Zone cache:
$JGTPY_DATA/zone/{instrument}_{timeframe}.json
```

---

## Dependencies

```python
import pandas as pd
from jgtpy import JGTIDS as ids
from jgtpy import JGTPDSP as pds
from jgtapy import Indicators
from jgtutils.jgtos import get_data_path, mk_fullpath
from jgtutils import jgtconstants as c
from JGTCDSRequest import JGTCDSRequest
from JGTChartConfig import JGTChartConfig
```

---

## Quality Criteria

✅ **Complete Williams Suite**: All 5 dimensions calculated  
✅ **FDB Signals**: Entry signals with fractal + AO confirmation  
✅ **Three Alligators**: Regular (5-8-13), Big (34-55-89), Tide (144-233-377)  
✅ **Zone Coloring**: Buying/Selling/Neutral zones identified  
✅ **MFI Analysis**: Market Facilitation Index classification  
✅ **Service Layer**: Programmatic API for integrations

===============================================
FILE: ids.spec.md
PATH: jgtpy/rispecs/ids.spec.md
URL:  https://jgtpy.jgwill.com/rispecs/ids.spec.md
===============================================

# IDS (Indicator Data Service) Specification

> Williams Trading System Indicator Calculations

**Specification Version**: 1.0  
**Module**: `jgtpy/JGTIDS.py`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: DataFrames enriched with complete Bill Williams Trading System indicators - the mathematical foundation for Chaos Trading signals.

**Achievement Indicator**: Given a PDS DataFrame, produces DataFrame with:
- All Alligator variants (Regular, Big, Tide)
- Awesome Oscillator (AO) with color
- Accelerator Oscillator (AC) with color
- Fractals at multiple periods (3, 5, 8, 13, 21, 34, 55, 89)
- FDB signals and Zero Line Crosses

**Value Proposition**: Single function to add all Williams indicators - the computational core of the trading system.

---

## Structural Tension

**Current Reality**: Raw OHLCV price data with no analytical value.

**Desired State**: Price data with complete indicator suite enabling Williams Trading System analysis.

**Natural Progression**: OHLCV → Alligator → AO/AC → Fractals → FDB → ZLC → Zone.

---

## Core Function: ids_add_indicators

```python
def ids_add_indicators(
    dfsrc: pd.DataFrame,
    dropnavalue: bool = True,
    quiet: bool = True,
    cleanupOriginalColumn: bool = True,
    useLEGACY: bool = True,
    cc: JGTChartConfig = None,
    bypass_index_reset: bool = False,
    rq: JGTIDSRequest = None
) -> pd.DataFrame:
    """
    Add all Williams Trading System indicators to DataFrame.
    
    Args:
        dfsrc: DataFrame with OHLCV columns
        dropnavalue: Drop rows with NaN values
        quiet: Suppress console output
        cleanupOriginalColumn: Remove intermediate columns
        cc: Chart configuration (optional)
        rq: Request object with indicator flags
    
    Calculation Order:
        1. Alligator (Regular 5-8-13)
        2. Big Alligator (34-55-89) - if enabled
        3. Tide Alligator (144-233-377) - if enabled
        4. Awesome Oscillator
        5. Accelerator Oscillator
        6. Gator Oscillator - if enabled
        7. Fractals (multiple periods)
        8. FDB (Fractal Divergent Bar) signals
        9. Zero Line Cross signals
        10. Zone coloring
        11. MFI (Market Facilitation Index) - if enabled
        12. AO/Price peaks - if enabled
    
    Returns:
        DataFrame with all indicators added
    """
```

---

## IDS Request Object

```python
@dataclass
class JGTIDSRequest:
    """Configuration for indicator calculations."""
    
    # Alligator flags
    gator_oscillator_flag: bool = False
    balligator_flag: bool = False
    talligator_flag: bool = False
    
    # Alligator periods
    balligator_period_jaws: int = 89   # Big: 34-55-89
    talligator_period_jaws: int = 377  # Tide: 144-233-377
    
    # Feature flags
    mfi_flag: bool = True
    ao_peaks_flag: bool = False
    mouth_water_flag: bool = False
    
    # Fractal configuration
    largest_fractal_period: int = 89
    
    # Processing options
    quotescount: int = 300
    keep_bid_ask: bool = True
    dropna_volume: bool = True
```

---

## Alligator Calculation

### Regular Alligator (5-8-13)

```python
# Columns: jaw, teeth, lips
median = (High + Low) / 2

jaw = SMMA(median, period=13).shift(8)     # Blue line
teeth = SMMA(median, period=8).shift(5)    # Red line
lips = SMMA(median, period=5).shift(3)     # Green line

# SMMA = Smoothed Moving Average
# First value: SMA(period)
# Subsequent: (prev_smma * (period - 1) + current) / period
```

### Big Alligator (34-55-89)

```python
# Columns: bjaw, bteeth, blips
bjaw = SMMA(median, period=89).shift(8)
bteeth = SMMA(median, period=55).shift(5)
blips = SMMA(median, period=34).shift(3)
```

### Tide Alligator (144-233-377)

```python
# Columns: tjaw, tteeth, tlips
tjaw = SMMA(median, period=377).shift(8)
tteeth = SMMA(median, period=233).shift(5)
tlips = SMMA(median, period=144).shift(3)
```

---

## Awesome Oscillator (AO)

```python
def calculate_ao(df: pd.DataFrame) -> pd.Series:
    """
    Awesome Oscillator: 5-period SMA minus 34-period SMA of median price.
    
    Formula:
        median = (High + Low) / 2
        AO = SMA(median, 5) - SMA(median, 34)
    
    Color:
        aocolor = "green" if AO[i] > AO[i-1] else "red"
    """
    median = (df['High'] + df['Low']) / 2
    ao = median.rolling(5).mean() - median.rolling(34).mean()
    return ao
```

---

## Accelerator Oscillator (AC)

```python
def calculate_ac(df: pd.DataFrame) -> pd.Series:
    """
    Accelerator Oscillator: AO minus 5-period SMA of AO.
    
    Formula:
        AC = AO - SMA(AO, 5)
    
    Color:
        accolor = "green" if AC[i] > AC[i-1] else "red"
    """
    ao = df['ao']
    ac = ao - ao.rolling(5).mean()
    return ac
```

---

## Gator Oscillator

```python
def calculate_gator(df: pd.DataFrame) -> Tuple[pd.Series, pd.Series]:
    """
    Gator Oscillator: Visual representation of Alligator convergence/divergence.
    
    Upper (Jaw-Teeth):
        gator_upper = abs(jaw - teeth)
        gator_upper_color = "green" if increasing else "red"
    
    Lower (Teeth-Lips):
        gator_lower = -abs(teeth - lips)  # Negative for display
        gator_lower_color = "green" if decreasing else "red"
    
    Phases:
        - Sleeping: Both bars decreasing
        - Awakening: One bar growing, one shrinking
        - Feeding: Both bars increasing
        - Sated: Both bars beginning to decrease
    """
```

---

## Fractal Calculation

```python
def calculate_fractals(
    df: pd.DataFrame,
    periods: List[int] = [3, 5, 8, 13, 21, 34, 55, 89]
) -> pd.DataFrame:
    """
    Calculate fractal highs and lows at multiple periods.
    
    Fractal High (period N):
        High[i] is highest of High[i-N//2:i+N//2+1]
        Must have N bars around it (centered)
    
    Fractal Low (period N):
        Low[i] is lowest of Low[i-N//2:i+N//2+1]
        Must have N bars around it (centered)
    
    Output Columns:
        fh, fl (5-bar default)
        fh3, fl3 (3-bar)
        fh5, fl5 (5-bar explicit)
        fh8, fl8 (8-bar)
        fh13, fl13 (13-bar)
        ...up to largest_fractal_period
    """
```

---

## FDB (Fractal Divergent Bar) Signals

```python
def calculate_fdb(df: pd.DataFrame) -> pd.DataFrame:
    """
    Fractal Divergent Bar - Primary entry signal.
    
    FDB Buy Conditions (fdbb=1):
        1. Fractal Low present (fl=1)
        2. Bar closes above Alligator Teeth
        3. Low is below Alligator Teeth
        4. AO supports direction (green preferred)
    
    FDB Sell Conditions (fdbs=1):
        1. Fractal High present (fh=1)
        2. Bar closes below Alligator Teeth
        3. High is above Alligator Teeth
        4. AO supports direction (red preferred)
    
    Combined Signal:
        fdb = fdbb - fdbs  # 1=buy, -1=sell, 0=none
    """
```

---

## Zero Line Cross (ZLC)

```python
def calculate_zlc(df: pd.DataFrame) -> pd.DataFrame:
    """
    Zero Line Cross detection for AO.
    
    ZLC Buy (zlcb=1):
        AO crosses from negative to positive
        AO[i-1] < 0 and AO[i] >= 0
    
    ZLC Sell (zlcs=1):
        AO crosses from positive to negative
        AO[i-1] > 0 and AO[i] <= 0
    
    Combined:
        zlc = zlcb - zlcs
    """
```

---

## Zone Calculation

```python
def calculate_zone(df: pd.DataFrame) -> pd.DataFrame:
    """
    Zone coloring based on AO and AC momentum.
    
    Buying Zone (bz=1, zcol="B"):
        - 2+ consecutive green AO bars
        - AND 2+ consecutive green AC bars
        - Strong bullish momentum
    
    Selling Zone (sz=1, zcol="S"):
        - 2+ consecutive red AO bars
        - AND 2+ consecutive red AC bars
        - Strong bearish momentum
    
    Neutral Zone (zcol="N"):
        - Neither buying nor selling zone
        - Choppy/consolidating market
    
    zone_sig: Same as zcol for consistency
    """
```

---

## MFI (Market Facilitation Index)

```python
def calculate_mfi(df: pd.DataFrame) -> pd.DataFrame:
    """
    Market Facilitation Index classification.
    
    Formula:
        MFI = (High - Low) / Volume
    
    Classification (comparing to previous bar):
        Green: MFI up, Volume up (strong trend)
        Fade: MFI down, Volume down (trend exhaustion)
        Squat: MFI down, Volume up (battle, potential reversal)
        Fake: MFI up, Volume down (false move)
    
    Columns:
        mfi: Raw MFI value
        mfi_sig: Classification letter (G/F/S/K)
    """
```

---

## Column Constants

```python
# Regular Alligator
JAW = "jaw"
TEETH = "teeth"
LIPS = "lips"

# Big Alligator
BJAW = "bjaw"
BTEETH = "bteeth"
BLIPS = "blips"

# Tide Alligator
TJAW = "tjaw"
TTEETH = "tteeth"
TLIPS = "tlips"

# Oscillators
AO = "ao"
AC = "ac"
AOCOLOR = "aocolor"
ACCOLOR = "accolor"

# Fractals
FH = "fh"
FL = "fl"
FH3, FL3 = "fh3", "fl3"
FH5, FL5 = "fh5", "fl5"
# ... up to FH89, FL89

# Signals
FDB = "fdb"
FDBB = "fdbb"
FDBS = "fdbs"
ZLC = "zlc"
ZLCB = "zlcb"
ZLCS = "zlcs"

# Zone
BZ = "bz"
SZ = "sz"
ZCOL = "zcol"
ZONE_SIGNAL = "zone_sig"

# MFI
MFI = "mfi"
MFI_SIG = "mfi_sig"
```

---

## Helper Functions

```python
def normalize_columns(
    df: pd.DataFrame,
    columns: List[str],
    in_place: bool = True
) -> pd.DataFrame:
    """Normalize columns to [-1, 1] range for ML."""

def _jgtpd_col_add_range_shifting(
    dfsrc: pd.DataFrame,
    ctxcolname: str,
    colprefix: str,
    endrange: int
) -> pd.DataFrame:
    """Add shifted column versions for lag features."""
```

---

## Dependencies

```python
import pandas as pd
import numpy as np
from jgtapy import Indicators           # Core indicator library
from jgtutils.jgtconstants import *     # Column name constants
from JGTIDSRequest import JGTIDSRequest
from JGTChartConfig import JGTChartConfig
from aohelper import add_ao_price_peaks_v2
from alligator_mouth_water import AlligatorMouthWaterAnalyzer  # Optional
```

---

## Quality Criteria

✅ **Complete Williams Suite**: All 5 dimensions implemented  
✅ **Three Alligators**: Regular, Big, Tide with correct periods  
✅ **Multi-period Fractals**: 3, 5, 8, 13, 21, 34, 55, 89 bars  
✅ **Proper Shifts**: Alligator lines shifted correctly  
✅ **Color Coding**: AO/AC colors for visual analysis  
✅ **Zone Detection**: Buying/Selling zone identification

===============================================
FILE: add-pattern.spec.md
PATH: jgtutils/rispecs/add-pattern.spec.md
URL:  https://jgtutils.jgwill.com/rispecs/add-pattern.spec.md
===============================================

# jgt-add-pattern (Pattern Manager) Specification

> Manage Column Patterns in User Settings

**Specification Version**: 1.0  
**Module**: `jgtutils/add_pattern_to_home_settings.py`  
**CLI Command**: `jgt-add-pattern`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Named column patterns stored in `~/.jgt/settings.json` for use by TTF and other data processing tools.

**Achievement Indicator**: Running `jgt-add-pattern --add-pattern mz --columns jaw teeth lips ao ac` produces:
- Pattern "mz" saved to settings
- Available for `ttfcli -pn mz`

**Value Proposition**: Define reusable column sets for cross-timeframe feature generation without repeating column lists.

---

## Structural Tension

**Current Reality**: User needs specific columns from higher timeframes but must specify them each time.

**Desired State**: Named patterns accessible across all CLI tools.

**Natural Progression**: Load settings → Add/Update/Remove pattern → Save settings.

---

## CLI Interface

```python
def main():
    """
    Manage patterns in home settings.
    
    Arguments (mutually exclusive):
        --add-pattern NAME: Name of pattern to add
        --list-patterns: List all patterns
        --remove-pattern NAME: Name of pattern to remove
    
    Additional arguments:
        --columns COL1 COL2 ...: Columns for pattern (required with --add-pattern)
        --force: Overwrite existing pattern
    
    Examples:
        # Add new pattern
        jgt-add-pattern --add-pattern mz --columns jaw teeth lips ao ac
        
        # List all patterns
        jgt-add-pattern --list-patterns
        
        # Remove pattern
        jgt-add-pattern --remove-pattern mz
        
        # Force overwrite existing
        jgt-add-pattern --add-pattern mz --columns jaw teeth lips --force
    """
```

---

## Core Functions

### Settings I/O

```python
def load_home_settings() -> Dict[str, Any]:
    """
    Load settings from ~/.jgt/settings.json.
    
    Returns empty structure if file doesn't exist:
        {"patterns": {}}
    """

def save_home_settings(settings: Dict[str, Any]) -> bool:
    """
    Save settings to ~/.jgt/settings.json.
    
    Creates ~/.jgt directory if needed.
    Returns True on success, False on failure.
    """
```

### Pattern Management

```python
def add_pattern_to_settings(
    pattern_name: str,
    columns: List[str],
    force: bool = False
) -> bool:
    """
    Add or update a pattern.
    
    Algorithm:
        1. Load settings
        2. If pattern exists and not force: return False
        3. Add pattern with columns list
        4. Save settings
    
    Storage format:
        {"patterns": {"mz": {"columns": ["jaw", "teeth", "lips", "ao", "ac"]}}}
    """

def list_patterns() -> None:
    """
    Print all patterns with their columns.
    
    Output:
        Patterns in home settings:
        ----------------------------------------
        mz: jaw, teeth, lips, ao, ac
        williams: ao, ac, bz, sz
    """

def remove_pattern_from_settings(pattern_name: str) -> bool:
    """
    Remove a pattern from settings.
    
    Returns True if removed, False if not found.
    """
```

---

## Settings Structure

```json
{
    "patterns": {
        "mz": {
            "columns": ["jaw", "teeth", "lips", "ao", "ac"]
        },
        "williams": {
            "columns": ["ao", "ac", "aocolor", "accolor", "bz", "sz", "fh", "fl"]
        },
        "alligator": {
            "columns": ["jaw", "teeth", "lips", "bjaw", "bteeth", "blips", "tjaw", "tteeth", "tlips"]
        }
    }
}
```

---

## File Locations

```python
# Settings file
~/.jgt/settings.json

# Directory created if missing
~/.jgt/
```

---

## Integration with TTF

```python
# In ttfcli.py
# Pattern name is used to select columns:
ttfcli -i EUR/USD -t H4 -pn mz

# TTF loads pattern and uses columns for HTF features:
# jaw, teeth, lips, ao, ac from H4 → added to H1 data
```

---

## Default Patterns

```python
# Common patterns (suggested for setup)
patterns = {
    "mz": ["jaw", "teeth", "lips", "ao", "ac"],
    "ttf": ["jaw", "teeth", "lips", "ao", "ac", "aocolor", "accolor"],
    "alligator_all": ["jaw", "teeth", "lips", "bjaw", "bteeth", "blips", "tjaw", "tteeth", "tlips"],
    "signals": ["fdb", "fdbb", "fdbs", "zlcB", "zlcS", "bz", "sz"]
}
```

---

## Dependencies

```python
import argparse
import json
import os
import sys
from typing import List, Dict, Any
from jgtutils import jgtcommon
```

---

## Quality Criteria

✅ **CRUD Operations**: Add, list, remove patterns  
✅ **Force Overwrite**: Prevent accidental overwrites  
✅ **Home Directory**: User-level persistence  
✅ **Auto Directory**: Creates ~/.jgt if needed  
✅ **Clean Output**: Formatted pattern list

===============================================
FILE: constants.spec.md
PATH: jgtutils/rispecs/constants.spec.md
URL:  https://jgtutils.jgwill.com/rispecs/constants.spec.md
===============================================

# jgtutils Constants Specification

> Column Names and System-Wide Constants

**Specification Version**: 1.0  
**Module**: `jgtutils/jgtconstants.py`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Consistent column naming across the entire JGT ecosystem, ensuring all packages speak the same language for data columns and indicator names.

**Achievement Indicator**: All packages import constants from jgtutils and use the same column names, preventing typos and mismatches in DataFrame operations.

**Value Proposition**: Single source of truth for column names, zone colors, and configuration defaults.

---

## Price Column Constants

```python
# Standard OHLCV
OPEN = "Open"
HIGH = "High"
LOW = "Low"
CLOSE = "Close"
VOLUME = "Volume"
DATE = "Date"
MEDIAN = "Median"
BAR_HEIGHT = "bar_height"

# Bid columns
BIDOPEN = "BidOpen"
BIDHIGH = "BidHigh"
BIDLOW = "BidLow"
BIDCLOSE = "BidClose"

# Ask columns
ASKOPEN = "AskOpen"
ASKHIGH = "AskHigh"
ASKLOW = "AskLow"
ASKCLOSE = "AskClose"
```

---

## Alligator Constants

### Regular Alligator (5-8-13)

```python
JAW = "jaw"      # 13-period SMMA, shift 8
TEETH = "teeth"  # 8-period SMMA, shift 5
LIPS = "lips"    # 5-period SMMA, shift 3
```

### Big Alligator (34-55-89)

```python
BJAW = "bjaw"
BTEETH = "bteeth"
BLIPS = "blips"

BJAW_PERIODS = 89
BTEETH_PERIODS = 55
BLIPS_PERIODS = 34
```

### Tide Alligator (144-233-377)

```python
TJAW = "tjaw"
TTEETH = "tteeth"
TLIPS = "tlips"

TJAW_PERIODS = 377
TTEETH_PERIODS = 233
TLIPS_PERIODS = 144
```

---

## Oscillator Constants

```python
# Awesome Oscillator
AO = "ao"
AOAZ = "aoaz"    # AO above zero
AOBZ = "aobz"    # AO below zero
AOCOLOR = "aocolor"

# Accelerator Oscillator
AC = "ac"
ACCOLOR = "accolor"

# Zero Line Cross
ZLC = "zlc"
ZLCB = "zlcb"    # ZLC Buy
ZLCS = "zlcs"    # ZLC Sell

# Gator Oscillator
GL = "gl"        # Gator low
GH = "gh"        # Gator high
```

---

## Fractal Constants

```python
# Standard fractals
FH = "fh"        # Fractal high (5-bar)
FL = "fl"        # Fractal low (5-bar)

# Multi-period fractals
FH3 = "fh3"
FL3 = "fl3"
FH5 = "fh5"
FL5 = "fl5"
FH8 = "fh8"
FL8 = "fl8"
FH13 = "fh13"
FL13 = "fl13"
FH21 = "fh21"
FL21 = "fl21"
FH34 = "fh34"
FL34 = "fl34"
FH55 = "fh55"
FL55 = "fl55"
FH89 = "fh89"
FL89 = "fl89"
```

---

## Signal Constants

```python
# FDB signals
FDB = "fdb"        # Combined: 1 (buy), -1 (sell), 0 (none)
FDBB = "fdbb"      # FDB Buy
FDBS = "fdbs"      # FDB Sell

# Target for ML
FDB_TARGET = "target"

# AO Vector counts
VECTOR_AO_FDBS = "vaos"
VECTOR_AO_FDBB = "vaob"
VECTOR_AO_FDBS_COUNT = "vaosc"
VECTOR_AO_FDBB_COUNT = "vaobc"
VECTOR_AO_FDB_COUNT = "vaoc"
```

---

## Zone Constants

```python
# Zone signals
BZ = "bz"              # Buying zone
SZ = "sz"              # Selling zone
ZCOL = "zcol"          # Zone color
ZONE_SIGNAL = "zone_sig"

# Zone colors
buyingZoneColor = "green"
sellingZoneColor = "red"
nonTradingZoneColor = "gray"
```

---

## MFI Constants

```python
MFI = "mfi"            # Market Facilitation Index
MFI_SQUAT = "mfi_sq"   # Squat signal
MFI_GREEN = "mfi_g"    # Green signal
MFI_FADE = "mfi_f"     # Fade signal
MFI_FAKE = "mfi_k"     # Fake signal
MFI_SIG = "mfi_sig"    # Signal classification
```

---

## Peak Detection Constants

```python
PRICE_PEAK_ABOVE = "price_peak_above"
PRICE_PEAK_BELLOW = "price_peak_bellow"
AO_PEAK_ABOVE = "ao_peak_above"
AO_PEAK_BELLOW = "ao_peak_bellow"
```

---

## Mouth Water Constants

```python
MW_LIP_TOUCH_BUY = "mw_ltb"
MW_LIP_TOUCH_SELL = "mw_lts"
MW_TEETH_TOUCH_BUY = "mw_ttb"
MW_TEETH_TOUCH_SELL = "mw_tts"
```

---

## Configuration Defaults

```python
# Default bar counts
NB_BARS_BY_DEFAULT = 335
NB_BARS_BY_DEFAULT_IN_CDS = 300

# ML default columns
ML_DEFAULT_COLUMNS_TO_KEEP = [
    'High', 'Low', 'ao', 'ac',
    'jaw', 'teeth', 'lips',
    'fh', 'fl', 'fdbb', 'fdbs',
    'zlcb', 'zlcs', 'target',
    'vaosc', 'vaobc'
]

# Columns to remove from output
columns_to_remove = [
    'aofvalue', 'aofhighao', 'aoflowao',
    'aofhigh', 'aoflow', 'aocolor', 'accolor',
    'fdbbhigh', 'fdbblow', 'fdbshigh', 'fdbslow'
]
```

---

## Type Definitions

```python
# Column type hints
AOAZ_TYPE = int
AOBZ_TYPE = int
ZLC_TYPE = int
```

---

## Usage Pattern

```python
from jgtutils.jgtconstants import (
    HIGH, LOW, CLOSE, OPEN,
    JAW, TEETH, LIPS,
    BJAW, BTEETH, BLIPS,
    FDB, FDBB, FDBS,
    AO, AC, ZLC
)

# Use constants for column access
entry_price = df[HIGH].iloc[-1]
is_buy_signal = df[FDBB].iloc[-1] == 1
alligator_direction = df[LIPS].iloc[-1] < df[TEETH].iloc[-1]
```

---

## Quality Criteria

✅ **Single Source**: All column names defined once  
✅ **Type Safety**: Type hints for key columns  
✅ **Complete Coverage**: All Williams indicators included  
✅ **Consistent Naming**: Lowercase for indicator columns  
✅ **Backward Compatible**: Legacy aliases maintained

===============================================
FILE: jgtset.spec.md
PATH: jgtutils/rispecs/jgtset.spec.md
URL:  https://jgtutils.jgwill.com/rispecs/jgtset.spec.md
===============================================

# jgtset (Settings Manager) Specification

> Export and Manage JGT Settings Across Formats

**Specification Version**: 1.0  
**Module**: `jgtutils/jgtset.py`  
**CLI Command**: `jgtset`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Exported settings in various formats (env vars, JSON, YAML) and synchronized configuration files.

**Achievement Indicator**: Running `jgtset -E` produces:
- Settings exported to `~/.jgt/env.sh`
- Ready for `source ~/.jgt/env.sh` in bash scripts

**Value Proposition**: Bridge JGT settings across shell environments, YAML config files, and programmatic access.

---

## Structural Tension

**Current Reality**: Settings live in `~/.jgt/settings.json` but need to be accessible in bash, YAML configs, and environment.

**Desired State**: Settings synchronized across all required formats.

**Natural Progression**: Load settings → Filter by keys → Format output → Export/update target.

---

## CLI Interface

```python
def main():
    """
    JGT Settings Export CLI.
    
    Arguments:
        -E, --export-env: Export settings as environment variables
        -K, --keys: Export only specified keys
        -J, --json: Print as JSON
        -Y, --yaml: Print as YAML
        -S, --silent: Silent output
        -V, --view: View settings list
        -O, --output: Specify output env file path
        -U, --update: Update existing YAML file with jgt settings
        -R, --reset-jgt-config: Reset YAML file's jgt section
        -ls, --settings: Load from specific settings file
    
    Examples:
        # Export all settings to env file
        jgtset -E
        
        # Export specific keys only
        jgtset -E -K instruments timeframes
        
        # Print as JSON
        jgtset -J
        
        # Print as YAML
        jgtset -Y
        
        # Update Jekyll _config.yml with jgt settings
        jgtset -U _config.yml jgt
        
        # View available settings keys
        jgtset -V
        
        # Export to custom file
        jgtset -E -O /tmp/my_env.sh
    """
```

---

## Core Functions

### Settings Loading

```python
def _load_settings(custom_path: str = None) -> dict:
    """
    Load settings from hierarchy:
        1. /etc/jgt/settings.json (system)
        2. ~/.jgt/settings.json (user)
        3. .jgt/settings.json (project)
        4. custom_path if provided
    
    Later files override earlier ones.
    """
    return jgtcommon.get_settings(custom_path=custom_path)
```

### Environment Export

```python
def export_keys_to_environ(
    _settings: dict = None,
    keys: List[str] = None,
    quiet: bool = True,
    env_file: str = None,
    custom_path: str = None
) -> None:
    """
    Export settings to environment file.
    
    Algorithm:
        1. Initialize env file with header
        2. Filter settings by keys if specified
        3. For each key-value pair:
           - Skip excluded keys (credentials, etc.)
           - Format lists as comma-separated quoted strings
           - Format bools as lowercase
           - Handle nested dicts recursively
           - Write to env file
    
    Output format:
        KEY=value
        LIST_KEY="item1,item2,item3"
        BOOL_KEY=true
    """
```

### JSON/YAML Output

```python
def dump_as_json_output(
    _settings: dict = None,
    keys: List[str] = None,
    custom_path: str = None
) -> str:
    """Return settings as formatted JSON string."""

def dump_as_yaml_output(
    _settings: dict = None,
    keys: List[str] = None,
    custom_path: str = None
) -> str:
    """Return settings as formatted YAML string."""
```

### YAML File Update

```python
def update_jgt_on_existing_yaml_file(
    target_filepath: str,
    _settings: dict = None,
    keys: List[str] = None,
    custom_path: str = None,
    target_key: str = 'jgt',
    add_on_only: bool = True
) -> dict:
    """
    Update existing YAML file with JGT settings.
    
    Algorithm:
        1. Load current YAML file
        2. Create backup (.bak)
        3. If add_on_only: merge only new keys
        4. Else: replace entire target_key section
        5. Remove excluded keys
        6. Save updated YAML
    
    Use cases:
        - Jekyll _config.yml integration
        - Docker compose settings
        - Ansible variable files
    """
```

---

## Settings Filtering

```python
# Excluded keys (never exported)
_JGTSET_EXCLUDED_ENV_EXPORT_KEYS = [
    "fx_user_id",
    "fx_password",
    "fx_url",
    "fx_connection",
    "fx_account"
]

# Optional included keys (from settings or env)
# jgtset_included: "instruments,timeframes,patterns"
# JGTSET_INCLUDED=instruments,timeframes,patterns
```

---

## List Formatting

```python
def __format_list_to_string(
    value: list,
    enquote: bool = True,
    single_quote: bool = False
) -> str:
    """
    Format list for shell export.
    
    Example:
        ["a", "b", "c"] -> '"a,b,c"'
        [1, 2, 3] -> '"1,2,3"'
    """
```

---

## Output File Location

```python
# Default env export path
~/.jgt/env.sh

# Structure
#!/bin/bash
# This file is generated by JGTSettingsCLI (jgtset)
instruments="EUR/USD,GBP/USD,USD/JPY"
timeframes="H1,H4,D1"
columns_to_remove="BidOpen,BidHigh,BidLow,BidClose,AskOpen,AskHigh,AskLow,AskClose"
```

---

## Integration Points

### jgtfxcon
- Read `instruments`, `timeframes` for batch operations
- Read `columns_to_remove` for data cleaning

### jgtml
- Read `patterns` for TTF column selection
- Read `ttf2run` for workflow configuration

### jgt-data-server
- Load settings at startup for API defaults

### jgt-code
- Access settings via MCP tools

---

## Dependencies

```python
import json
import ruamel.yaml  # Optional - graceful fallback
import dotenv
from jgtutils import jgtcommon
from jgtutils.jgtenv import get_dotenv_jgtset_export_path
```

---

## Quality Criteria

✅ **Multi-Format**: JSON, YAML, and env var export  
✅ **Selective Export**: Filter by key names  
✅ **YAML Integration**: Update existing config files  
✅ **Credential Protection**: Excluded keys never exported  
✅ **List Handling**: Proper shell-compatible formatting  
✅ **Hierarchical Loading**: System → User → Project override

===============================================
FILE: jgtutr.spec.md
PATH: jgtutils/rispecs/jgtutr.spec.md
URL:  https://jgtutils.jgwill.com/rispecs/jgtutr.spec.md
===============================================

# jgtutr (TLID Range Calculator) Specification

> Timeline ID Range Calculator for Date-based Data Queries

**Specification Version**: 1.0  
**Module**: `jgtutils/cli_tlid_range.py`  
**CLI Command**: `jgtutr`  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: TLID (Timeline ID) range strings for querying historical price data over specific date periods.

**Achievement Indicator**: Running `jgtutr -e 2026-01-31 -t H4 -c 100` produces:
- TLID range string for 100 H4 periods ending at given date
- Format suitable for passing to data query CLIs

**Value Proposition**: Automatically calculate date ranges for backtesting and historical data retrieval without manual date math.

---

## Structural Tension

**Current Reality**: User knows "last 100 H4 bars ending today" but needs specific date range.

**Desired State**: Precise TLID range string usable by other CLI tools.

**Natural Progression**: Parse args → Calculate period duration → Subtract from end date → Format output.

---

## CLI Interface

```python
def main():
    """
    Calculate TLID range for data queries.
    
    Arguments:
        -e: End datetime (ISO format or descriptive)
        -t: Timeframe (m1, m5, m15, m30, H1, H4, D1, W1, MN)
        -c: Number of periods/bars to include
    
    Examples:
        # Last 100 H4 bars
        jgtutr -e 2026-01-31 -t H4 -c 100
        
        # Last 500 daily bars
        jgtutr -e 2026-01-31 -t D1 -c 500
        
        # 1000 M15 bars ending at specific time
        jgtutr -e "2026-01-31T12:00" -t m15 -c 1000
    
    Output:
        TLID range string formatted for -tlid argument
    """
```

---

## Core Function

```python
from jgtpov import calculate_tlid_range as get_tlid_range

def main():
    # Parse arguments
    args = parser.parse_args()
    
    # Calculate TLID range
    result = get_tlid_range(args.e, args.t, args.c)
    
    # Print result
    print(result)
```

---

## TLID Range Format

```python
# Output format
"{start_tlid}_{end_tlid}"

# Example
"260101_260131"  # Jan 1-31, 2026

# Used in other CLIs
jgtfxcli -i EUR/USD -t H4 --tlidrange 260101_260131
```

---

## Timeframe Duration Mapping

| Timeframe | Minutes | Used For |
|-----------|---------|----------|
| m1 | 1 | High-frequency analysis |
| m5 | 5 | Short-term scalping |
| m15 | 15 | Intraday trading |
| m30 | 30 | Intraday swings |
| H1 | 60 | Day trading |
| H4 | 240 | Swing trading |
| D1 | 1440 | Position trading |
| W1 | 10080 | Long-term trends |
| MN | 43200 | Macro analysis |

---

## Integration Points

- **jgtfxcli**: Pass TLID range for historical data fetch
- **cdscli**: Generate signals for specific date range
- **ttfcli**: Process cross-TF features for period
- **jgtapp**: Used in backtesting workflows

---

## Dependencies

```python
from jgtpov import calculate_tlid_range
import argparse
```

---

## Quality Criteria

✅ **Timeframe Aware**: Correct period calculation per TF  
✅ **Flexible Dates**: Multiple input formats supported  
✅ **CLI Ready**: Output suitable for other tools  
✅ **Simple Interface**: Just 3 arguments

===============================================
FILE: tfw-wtf.spec.md
PATH: jgtutils/rispecs/tfw-wtf.spec.md
URL:  https://jgtutils.jgwill.com/rispecs/tfw-wtf.spec.md
===============================================

# tfw/wtf (Timeframe Scheduler) Specification

> Production Trading Automation - Wait and Execute on Timeframe

**Specification Version**: 1.0  
**Module**: `jgtutils/timeframe_scheduler.py`  
**CLI Commands**: `tfw`, `wtf` (aliases)  
**RISE Framework Compliance**: Full  
**Last Updated**: 2026-01-31

---

## Desired Outcome Definition

**What Users Create**: Time-synchronized execution of scripts, CLIs, or functions aligned with trading timeframes.

**Achievement Indicator**: Running `tfw -t H1 -S /path/to/refresh.sh` produces:
- Process waits until H1 bar close (01:00, 02:00, etc.)
- Script executes at precise timeframe boundary
- Loop continues indefinitely until terminated

**Value Proposition**: Automate trading workflows to execute exactly when new candles close, ensuring fresh data and proper timing.

---

## Structural Tension

**Current Reality**: Need to run trading logic at specific timeframe intervals but cron lacks precision for varying DST and market schedules.

**Desired State**: Automated execution precisely at candle close for any supported timeframe.

**Natural Progression**: Parse args → Calculate trigger times → Wait loop → Execute action → Repeat.

---

## CLI Interface

```python
def main():
    """
    JGT Timeframe Scheduler (tfw/wtf).
    
    Arguments:
        -t, --timeframe: Target timeframe (m1, m5, m15, m30, H1, H4, D1, W1, M1) [required]
        -X, --exit: Exit when timeframe is reached (one-shot)
        -S, -B, --script-to-run: Bash script to execute at timeframe
        -C, --cli-to-run: CLI command to execute at timeframe
        -F, --function: Bash function to run (requires load.sh)
        -M, --message: Message to display when timeframe reached
        -I, --in-message: Message to display when wait starts
        -N, --no-output: Suppress all output
        -v, --verbose: Verbosity level
    
    Examples:
        # Wait for H1 then exit
        tfw -t H1 -X
        
        # Run script every m15
        tfw -t m15 -S /opt/trading/refresh.sh
        
        # Execute CLI on H4
        wtf -t H4 -C python trading_pipeline.py
        
        # Run bash function from load.sh
        tfw -t D1 -F daily_report
    """
```

---

## Core Algorithm

```python
def main():
    """
    Main scheduler loop.
    
    Flow:
        1. Parse arguments
        2. Validate timeframe from args or JGT_TIMEFRAME env
        3. Get trigger times for timeframe
        4. Calculate sleep duration (60s for >m1, 2s for m1)
        5. Print start message (unless -N)
        6. Infinite loop:
           a. Get current time
           b. If current_time in trigger_times:
              - If --exit: print message and exit
              - If --script: run script with timeframe, time as args
              - If --cli: run CLI command
              - Else: call refreshTF()
              - Adjust sleep for execution duration
           c. Sleep 1 second
    """
```

---

## Timeframe Trigger Calculation

```python
def get_times_by_timeframe_str(timeframe: str) -> List[str]:
    """
    Generate all trigger times for a timeframe.
    
    Returns:
        List of "HH:MM" (or "HH:MM:SS" for m1) strings
    """

def get_timeframes_times_by_minutes(minutes: int) -> List[str]:
    """
    Calculate trigger times by minute interval.
    
    For minutes >= 60: start at 01:00 (forex new day)
    For minutes < 60: start at 00:00
    For m1: includes :00 and :01 seconds (catches late triggers)
    
    Examples:
        H4: ["01:00", "05:00", "09:00", "13:00", "17:00", "21:00"]
        H1: ["01:00", "02:00", ..., "23:00"]
        m15: ["00:00", "00:15", "00:30", "00:45", "01:00", ...]
        m1: ["00:00:00", "00:00:01", "00:01:00", "00:01:01", ...]
    """

def get_timeframe_daily_ending_time() -> str:
    """
    Calculate D1/W1/MN trigger time based on DST.
    
    Returns:
        "22:00:00" during DST (March-November)
        "21:00:00" outside DST
    
    DST calculation:
        Start: Second Sunday in March
        End: First Sunday in November
    """
```

---

## Script Execution

```python
def _run_script_to_run(script_to_run: List[str]) -> None:
    """
    Execute script with context arguments.
    
    Command: bash {script_path} {timeframe} {current_time} {extra_args...}
    
    Example:
        bash /opt/trading/refresh.sh H4 13:00
    
    Script receives:
        $1 = timeframe
        $2 = trigger time
        $3+ = extra arguments from CLI
    """

def _run_function(
    function_to_run: str,
    load_script: str = "/opt/binscripts/load.sh"
) -> None:
    """
    Execute bash function with sourced environment.
    
    Load script search order:
        1. .jgt/load.sh (current dir)
        2. ../.jgt/load.sh (parent dir)
        3. ~/.jgt/load.sh (home)
        4. /opt/binscripts/load.sh (system)
    
    Command: bash -c ". {load_script} && {function} {timeframe} {time}"
    """
```

---

## JSONL Output

```python
def _print_app_message(
    timeframe: str,
    msg: str,
    state: str = None,
    use_short: bool = True
) -> None:
    """
    Print structured JSONL message.
    
    Format:
        {"message": "...", "timeframe": "H4", "time": "13:00", "state": "reached"}
    
    States:
        - started: Wait initiated
        - reached: Timeframe triggered
        - error: Execution failed
        - canceled: User interrupted
    """
```

---

## Error Handling

```python
# Exit codes
DOTJGTENV_TIMEFRAME_NOT_FOUND_EXIT_ERROR_CODE = 51
SUBPROCESS_RUN_ERROR_EXIT_ERROR_CODE = 52
BASH_FUNCTION_RUN_EXIT_ERROR_CODE = 53
BASH_LOADER_ERROR_EXIT_ERROR_CODE = 54

# Behavior
# - Script failures: log error, continue loop (unless exit_on_error)
# - CLI failures: log error, continue loop
# - Function failures: exit if exit_on_error=True
```

---

## Integration with jgtapp

```python
# jgtapp.py wrapper function
def w(
    timeframe: str,
    script_to_run: str = None,
    exit_on_timeframe: bool = False
) -> None:
    """
    Wrapper for tfw in jgtapp.
    
    Called by:
        - jgtapp w -t H4 -X
        - fxmvstopgator loop action
        - Trading campaign automation
    """
```

---

## Environment Variables

```python
# Load timeframe from env if not provided
JGT_TIMEFRAME = "H4"

# Custom load script location
JGT_LOAD_SCRIPT = "/path/to/load.sh"
```

---

## Dependencies

```python
import datetime
import time
import subprocess
from jgtutils import jgtcommon
from jgtutils.jgtclihelper import print_jsonl_message
import jgtwslhelper as wsl
```

---

## Quality Criteria

✅ **DST Aware**: Correct D1/W1 timing across seasons  
✅ **Multiple Actions**: Script, CLI, function, or exit  
✅ **Execution Timing**: Adjusts sleep for script duration  
✅ **Structured Output**: JSONL for log parsing  
✅ **Graceful Errors**: Continues on failures  
✅ **m1 Precision**: Catches late triggers with :01 variant

===============================================
END OF DOCUMENT