cli/pkg/ansible/parser.go
Snider 5e2765fd5f feat: wire release command, add tar.xz support, unified installers (#277)
* feat(cli): wire release command and add installer scripts

- Wire up `core build release` subcommand (was orphaned)
- Wire up `core monitor` command (missing import in full variant)
- Add installer scripts for Unix (.sh) and Windows (.bat)
  - setup: Interactive with variant selection
  - ci: Minimal for CI/CD environments
  - dev: Full development variant
  - go/php/agent: Targeted development variants
- All scripts include security hardening:
  - Secure temp directories (mktemp -d)
  - Architecture validation
  - Version validation after GitHub API call
  - Proper cleanup on exit
  - PowerShell PATH updates on Windows (avoids setx truncation)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(build): add tar.xz support and unified installer scripts

- Add tar.xz archive support using Borg's compress package
  - ArchiveXZ() and ArchiveWithFormat() for configurable compression
  - Better compression ratio than gzip for release artifacts
- Consolidate 12 installer scripts into 2 unified scripts
  - install.sh and install.bat with BunnyCDN edge variable support
  - Subdomains: setup.core.help, ci.core.help, dev.core.help, etc.
  - MODE and VARIANT transformed at edge based on subdomain
- Installers prefer tar.xz with automatic fallback to tar.gz
- Fixed CodeRabbit issues: HTTP status patterns, tar error handling,
  verify_install params, VARIANT validation, CI PATH persistence

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: add build and release config files

- .core/build.yaml - cross-platform build configuration
- .core/release.yaml - release workflow configuration

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: move plans from docs/ to tasks/

Consolidate planning documents in tasks/plans/ directory.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(install): address CodeRabbit review feedback

- Add curl timeout (--max-time) to prevent hanging on slow networks
- Rename TMPDIR to WORK_DIR to avoid clobbering system env var
- Add chmod +x to ensure binary has execute permissions
- Add error propagation after subroutine calls in batch file
- Remove System32 install attempt in CI mode (use consistent INSTALL_DIR)
- Fix HTTP status regex for HTTP/2 compatibility

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(rag): add Go RAG implementation with Qdrant + Ollama

Add RAG (Retrieval Augmented Generation) tools for storing documentation
in Qdrant vector database and querying with semantic search. This replaces
the Python tools/rag implementation with a native Go solution.

New commands:
- core rag ingest [directory] - Ingest markdown files into Qdrant
- core rag query [question] - Query vector database with semantic search
- core rag collections - List and manage Qdrant collections

Features:
- Markdown chunking by sections and paragraphs with overlap
- UTF-8 safe text handling for international content
- Automatic category detection from file paths
- Multiple output formats: text, JSON, LLM context injection
- Environment variable support for host configuration

Dependencies:
- github.com/qdrant/go-client (gRPC client)
- github.com/ollama/ollama/api (embeddings API)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(deploy): add pure-Go Ansible executor and Coolify API integration

Implement infrastructure deployment system with:

- pkg/ansible: Pure Go Ansible executor
  - Playbook/inventory parsing (types.go, parser.go)
  - Full execution engine with variable templating, loops, blocks,
    conditionals, handlers, and fact gathering (executor.go)
  - SSH client with key/password auth and privilege escalation (ssh.go)
  - 35+ module implementations: shell, command, copy, template, file,
    apt, service, systemd, user, group, git, docker_compose, etc. (modules.go)

- pkg/deploy/coolify: Coolify API client wrapping Python swagger client
  - List/get servers, projects, applications, databases, services
  - Generic Call() for any OpenAPI operation

- pkg/deploy/python: Embedded Python runtime for swagger client integration

- internal/cmd/deploy: CLI commands
  - core deploy servers/projects/apps/databases/services/team
  - core deploy call <operation> [params-json]

This enables Docker-free infrastructure deployment with Ansible-compatible
playbooks executed natively in Go.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(deploy): address linter warnings and build errors

- Fix fmt.Sprintf format verb error in ssh.go (remove unused stat command)
- Fix errcheck warnings by explicitly ignoring best-effort operations
- Fix ineffassign warning in cmd_ansible.go

All golangci-lint checks now pass for deploy packages.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* style(deploy): fix gofmt formatting

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(deploy): use known_hosts for SSH host key verification

Address CodeQL security alert by using the user's known_hosts file
for SSH host key verification when available. Falls back to accepting
any key only when known_hosts doesn't exist (common in containerized
or ephemeral environments).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(ai,security,ide): add agentic MVP, security jobs, and Core IDE desktop app

Wire up AI infrastructure with unified pkg/ai package (metrics JSONL,
RAG integration), move RAG under `core ai rag`, add `core ai metrics`
command, and enrich task context with Qdrant documentation.

Add `--target` flag to all security commands for external repo scanning,
`core security jobs` for distributing findings as GitHub Issues, and
consistent error logging across scan/deps/alerts/secrets commands.

Add Core IDE Wails v3 desktop app with Angular 20 frontend, MCP bridge
(loopback-only HTTP server), WebSocket hub, and Claude Code bridge.
Production-ready with Lethean CIC branding, macOS code signing support,
and security hardening (origin validation, body size limits, URL scheme
checks, memory leak prevention, XSS mitigation).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: address PR review comments from CodeRabbit, Copilot, and Gemini

Fixes across 25 files addressing 46+ review comments:

- pkg/ai/metrics.go: handle error from Close() on writable file handle
- pkg/ansible: restore loop vars after loop, restore become settings,
  fix Upload with become=true and no password (use sudo -n), honour
  SSH timeout config, use E() helper for contextual errors, quote git
  refs in checkout commands
- pkg/rag: validate chunk config, guard negative-to-uint64 conversion,
  use E() helper for errors, add context timeout to Ollama HTTP calls
- pkg/deploy/python: fix exec.ExitError type assertion (was os.PathError),
  handle os.UserHomeDir() error
- pkg/build/buildcmd: use cmd.Context() instead of context.Background()
  for proper Ctrl+C cancellation
- install.bat: add curl timeouts, CRLF line endings, use --connect-timeout
  for archive downloads
- install.sh: use absolute path for version check in CI mode
- tools/rag: fix broken ingest.py function def, escape HTML in query.py,
  pin qdrant-client version, add markdown code block languages
- internal/cmd/rag: add chunk size validation, env override handling

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(build): make release dry-run by default and remove darwin/amd64 target

Replace --dry-run (default false) with --we-are-go-for-launch (default
false) so `core build release` is safe by default. Remove darwin/amd64
from default build targets (arm64 only for macOS). Fix cmd_project.go
to use command context instead of context.Background().

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-04 00:49:57 +00:00

438 lines
10 KiB
Go

package ansible
import (
"fmt"
"os"
"path/filepath"
"strings"
"github.com/host-uk/core/pkg/log"
"gopkg.in/yaml.v3"
)
// Parser handles Ansible YAML parsing.
type Parser struct {
basePath string
vars map[string]any
}
// NewParser creates a new Ansible parser.
func NewParser(basePath string) *Parser {
return &Parser{
basePath: basePath,
vars: make(map[string]any),
}
}
// ParsePlaybook parses an Ansible playbook file.
func (p *Parser) ParsePlaybook(path string) ([]Play, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("read playbook: %w", err)
}
var plays []Play
if err := yaml.Unmarshal(data, &plays); err != nil {
return nil, fmt.Errorf("parse playbook: %w", err)
}
// Process each play
for i := range plays {
if err := p.processPlay(&plays[i]); err != nil {
return nil, fmt.Errorf("process play %d: %w", i, err)
}
}
return plays, nil
}
// ParseInventory parses an Ansible inventory file.
func (p *Parser) ParseInventory(path string) (*Inventory, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("read inventory: %w", err)
}
var inv Inventory
if err := yaml.Unmarshal(data, &inv); err != nil {
return nil, fmt.Errorf("parse inventory: %w", err)
}
return &inv, nil
}
// ParseTasks parses a tasks file (used by include_tasks).
func (p *Parser) ParseTasks(path string) ([]Task, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("read tasks: %w", err)
}
var tasks []Task
if err := yaml.Unmarshal(data, &tasks); err != nil {
return nil, fmt.Errorf("parse tasks: %w", err)
}
for i := range tasks {
if err := p.extractModule(&tasks[i]); err != nil {
return nil, fmt.Errorf("task %d: %w", i, err)
}
}
return tasks, nil
}
// ParseRole parses a role and returns its tasks.
func (p *Parser) ParseRole(name string, tasksFrom string) ([]Task, error) {
if tasksFrom == "" {
tasksFrom = "main.yml"
}
// Search paths for roles (in order of precedence)
searchPaths := []string{
// Relative to playbook
filepath.Join(p.basePath, "roles", name, "tasks", tasksFrom),
// Parent directory roles
filepath.Join(filepath.Dir(p.basePath), "roles", name, "tasks", tasksFrom),
// Sibling roles directory
filepath.Join(p.basePath, "..", "roles", name, "tasks", tasksFrom),
// playbooks/roles pattern
filepath.Join(p.basePath, "playbooks", "roles", name, "tasks", tasksFrom),
// Common DevOps structure
filepath.Join(filepath.Dir(filepath.Dir(p.basePath)), "roles", name, "tasks", tasksFrom),
}
var tasksPath string
for _, sp := range searchPaths {
// Clean the path to resolve .. segments
sp = filepath.Clean(sp)
if _, err := os.Stat(sp); err == nil {
tasksPath = sp
break
}
}
if tasksPath == "" {
return nil, log.E("parser.ParseRole", fmt.Sprintf("role %s not found in search paths: %v", name, searchPaths), nil)
}
// Load role defaults
defaultsPath := filepath.Join(filepath.Dir(filepath.Dir(tasksPath)), "defaults", "main.yml")
if data, err := os.ReadFile(defaultsPath); err == nil {
var defaults map[string]any
if yaml.Unmarshal(data, &defaults) == nil {
for k, v := range defaults {
if _, exists := p.vars[k]; !exists {
p.vars[k] = v
}
}
}
}
// Load role vars
varsPath := filepath.Join(filepath.Dir(filepath.Dir(tasksPath)), "vars", "main.yml")
if data, err := os.ReadFile(varsPath); err == nil {
var roleVars map[string]any
if yaml.Unmarshal(data, &roleVars) == nil {
for k, v := range roleVars {
p.vars[k] = v
}
}
}
return p.ParseTasks(tasksPath)
}
// processPlay processes a play and extracts modules from tasks.
func (p *Parser) processPlay(play *Play) error {
// Merge play vars
for k, v := range play.Vars {
p.vars[k] = v
}
for i := range play.PreTasks {
if err := p.extractModule(&play.PreTasks[i]); err != nil {
return fmt.Errorf("pre_task %d: %w", i, err)
}
}
for i := range play.Tasks {
if err := p.extractModule(&play.Tasks[i]); err != nil {
return fmt.Errorf("task %d: %w", i, err)
}
}
for i := range play.PostTasks {
if err := p.extractModule(&play.PostTasks[i]); err != nil {
return fmt.Errorf("post_task %d: %w", i, err)
}
}
for i := range play.Handlers {
if err := p.extractModule(&play.Handlers[i]); err != nil {
return fmt.Errorf("handler %d: %w", i, err)
}
}
return nil
}
// extractModule extracts the module name and args from a task.
func (p *Parser) extractModule(task *Task) error {
// First, unmarshal the raw YAML to get all keys
// This is a workaround since we need to find the module key dynamically
// Handle block tasks
for i := range task.Block {
if err := p.extractModule(&task.Block[i]); err != nil {
return err
}
}
for i := range task.Rescue {
if err := p.extractModule(&task.Rescue[i]); err != nil {
return err
}
}
for i := range task.Always {
if err := p.extractModule(&task.Always[i]); err != nil {
return err
}
}
return nil
}
// UnmarshalYAML implements custom YAML unmarshaling for Task.
func (t *Task) UnmarshalYAML(node *yaml.Node) error {
// First decode known fields
type rawTask Task
var raw rawTask
// Create a map to capture all fields
var m map[string]any
if err := node.Decode(&m); err != nil {
return err
}
// Decode into struct
if err := node.Decode(&raw); err != nil {
return err
}
*t = Task(raw)
t.raw = m
// Find the module key
knownKeys := map[string]bool{
"name": true, "register": true, "when": true, "loop": true,
"loop_control": true, "vars": true, "environment": true,
"changed_when": true, "failed_when": true, "ignore_errors": true,
"no_log": true, "become": true, "become_user": true,
"delegate_to": true, "run_once": true, "tags": true,
"block": true, "rescue": true, "always": true, "notify": true,
"retries": true, "delay": true, "until": true,
"include_tasks": true, "import_tasks": true,
"include_role": true, "import_role": true,
"with_items": true, "with_dict": true, "with_file": true,
}
for key, val := range m {
if knownKeys[key] {
continue
}
// Check if this is a module
if isModule(key) {
t.Module = key
t.Args = make(map[string]any)
switch v := val.(type) {
case string:
// Free-form args (e.g., shell: echo hello)
t.Args["_raw_params"] = v
case map[string]any:
t.Args = v
case nil:
// Module with no args
default:
t.Args["_raw_params"] = v
}
break
}
}
// Handle with_items as loop
if items, ok := m["with_items"]; ok && t.Loop == nil {
t.Loop = items
}
return nil
}
// isModule checks if a key is a known module.
func isModule(key string) bool {
for _, m := range KnownModules {
if key == m {
return true
}
// Also check without ansible.builtin. prefix
if strings.HasPrefix(m, "ansible.builtin.") {
if key == strings.TrimPrefix(m, "ansible.builtin.") {
return true
}
}
}
// Accept any key with dots (likely a module)
return strings.Contains(key, ".")
}
// NormalizeModule normalizes a module name to its canonical form.
func NormalizeModule(name string) string {
// Add ansible.builtin. prefix if missing
if !strings.Contains(name, ".") {
return "ansible.builtin." + name
}
return name
}
// GetHosts returns hosts matching a pattern from inventory.
func GetHosts(inv *Inventory, pattern string) []string {
if pattern == "all" {
return getAllHosts(inv.All)
}
if pattern == "localhost" {
return []string{"localhost"}
}
// Check if it's a group name
hosts := getGroupHosts(inv.All, pattern)
if len(hosts) > 0 {
return hosts
}
// Check if it's a specific host
if hasHost(inv.All, pattern) {
return []string{pattern}
}
// Handle patterns with : (intersection/union)
// For now, just return empty
return nil
}
func getAllHosts(group *InventoryGroup) []string {
if group == nil {
return nil
}
var hosts []string
for name := range group.Hosts {
hosts = append(hosts, name)
}
for _, child := range group.Children {
hosts = append(hosts, getAllHosts(child)...)
}
return hosts
}
func getGroupHosts(group *InventoryGroup, name string) []string {
if group == nil {
return nil
}
// Check children for the group name
if child, ok := group.Children[name]; ok {
return getAllHosts(child)
}
// Recurse
for _, child := range group.Children {
if hosts := getGroupHosts(child, name); len(hosts) > 0 {
return hosts
}
}
return nil
}
func hasHost(group *InventoryGroup, name string) bool {
if group == nil {
return false
}
if _, ok := group.Hosts[name]; ok {
return true
}
for _, child := range group.Children {
if hasHost(child, name) {
return true
}
}
return false
}
// GetHostVars returns variables for a specific host.
func GetHostVars(inv *Inventory, hostname string) map[string]any {
vars := make(map[string]any)
// Collect vars from all levels
collectHostVars(inv.All, hostname, vars)
return vars
}
func collectHostVars(group *InventoryGroup, hostname string, vars map[string]any) bool {
if group == nil {
return false
}
// Check if host is in this group
found := false
if host, ok := group.Hosts[hostname]; ok {
found = true
// Apply group vars first
for k, v := range group.Vars {
vars[k] = v
}
// Then host vars
if host != nil {
if host.AnsibleHost != "" {
vars["ansible_host"] = host.AnsibleHost
}
if host.AnsiblePort != 0 {
vars["ansible_port"] = host.AnsiblePort
}
if host.AnsibleUser != "" {
vars["ansible_user"] = host.AnsibleUser
}
if host.AnsiblePassword != "" {
vars["ansible_password"] = host.AnsiblePassword
}
if host.AnsibleSSHPrivateKeyFile != "" {
vars["ansible_ssh_private_key_file"] = host.AnsibleSSHPrivateKeyFile
}
if host.AnsibleConnection != "" {
vars["ansible_connection"] = host.AnsibleConnection
}
for k, v := range host.Vars {
vars[k] = v
}
}
}
// Check children
for _, child := range group.Children {
if collectHostVars(child, hostname, vars) {
// Apply this group's vars (parent vars)
for k, v := range group.Vars {
if _, exists := vars[k]; !exists {
vars[k] = v
}
}
found = true
}
}
return found
}