Commit graph

10 commits

Author SHA1 Message Date
Snider
6d65b70e0c Add streaming API to pkg/io and optimize agentic context gathering (#313)
* feat(io): add streaming API to Medium interface and optimize agentic context

- Added ReadStream and WriteStream to io.Medium interface.
- Implemented streaming methods in local and mock mediums.
- Updated pkg/agentic/context.go to use streaming I/O with LimitReader.
- Added 5000-byte truncation limit for all AI context file reads to reduce memory usage.
- Documented when to use streaming vs full-file APIs in io.Medium.

* feat(io): optimize streaming API and fix PR feedback

- Fixed resource leak in agentic context by using defer for closing file streams.
- Improved truncation logic in agentic context to handle multibyte characters correctly by checking byte length before string conversion.
- Added comprehensive documentation to ReadStream and WriteStream in local medium.
- Added unit tests for ReadStream and WriteStream in local medium.
- Applied formatting and fixed auto-merge CI configuration.

* feat(io): add streaming API and fix CI failures (syntax fix)

- Introduced ReadStream and WriteStream to io.Medium interface.
- Implemented streaming methods in local and mock mediums.
- Optimized agentic context with streaming reads and truncation logic.
- Fixed syntax error in local client tests by overwriting the file.
- Fixed auto-merge CI by adding checkout and repository context.
- Applied formatting fixes.
2026-02-05 11:00:49 +00:00
Snider
9f4007c409 Remove StrictHostKeyChecking=no from SSH commands (#315)
* Remove StrictHostKeyChecking=no and implement proper host key verification

This commit addresses security concerns from the OWASP audit by enforcing
strict host key verification for all SSH and SCP commands.

Key changes:
- Replaced StrictHostKeyChecking=accept-new with yes in pkg/container and pkg/devops.
- Removed insecure host key verification from pkg/ansible SSH client.
- Implemented a synchronous host key discovery mechanism during VM boot
  using ssh-keyscan to populate ~/.core/known_hosts.
- Updated the devops Boot lifecycle to wait until the host key is verified.
- Ensured pkg/ansible correctly handles missing known_hosts files.
- Refactored hardcoded SSH port 2222 to a package constant DefaultSSHPort.
- Added CORE_SKIP_SSH_SCAN environment variable for test environments.

* Remove StrictHostKeyChecking=no and implement proper host key verification

Addresses security concerns from OWASP audit by enforcing strict host key
verification.

Changes:
- Replaced StrictHostKeyChecking=accept-new with yes in pkg/container and devops.
- Removed insecure host key verification from pkg/ansible.
- Added synchronous host key discovery using ssh-keyscan during VM boot.
- Updated Boot lifecycle to wait for host key verification.
- Handled missing known_hosts file in pkg/ansible.
- Refactored hardcoded SSH port to DefaultSSHPort constant.
- Fixed formatting issues identified by QA check.

* Secure SSH commands and fix auto-merge CI failure

Addresses OWASP security audit by enforcing strict host key verification
and fixes a CI failure in the auto-merge workflow.

Key changes:
- Replaced StrictHostKeyChecking=accept-new with yes in pkg/container and pkg/devops.
- Removed insecure host key verification from pkg/ansible.
- Implemented synchronous host key discovery using ssh-keyscan during VM boot.
- Handled missing known_hosts file in pkg/ansible.
- Refactored hardcoded SSH port to DefaultSSHPort constant.
- Added pkg/ansible/ssh_test.go to verify SSH client initialization.
- Fixed formatting in pkg/io/local/client.go.
- Fixed auto-merge.yml by inlining the script and providing repository context
  to 'gh' command, resolving the "not a git repository" error in CI.

* Secure SSH, fix CI auto-merge, and resolve merge conflicts

This commit addresses the OWASP security audit by enforcing strict host key
verification and resolves persistent CI issues.

Security Changes:
- Replaced StrictHostKeyChecking=accept-new with yes in pkg/container and devops.
- Removed insecure host key verification from pkg/ansible.
- Implemented synchronous host key discovery using ssh-keyscan during VM boot.
- Updated Boot lifecycle to wait for host key verification.
- Handled missing known_hosts file in pkg/ansible.
- Refactored hardcoded SSH port to DefaultSSHPort constant.

CI and Maintenance:
- Fixed auto-merge.yml by inlining the script and adding repository context
  to 'gh' command, resolving the "not a git repository" error in CI.
- Resolved merge conflicts in .github/workflows/auto-merge.yml with dev branch.
- Added pkg/ansible/ssh_test.go for SSH client verification.
- Fixed formatting in pkg/io/local/client.go to pass QA checks.

* Secure SSH and TLS connections, and fix CI issues

Addresses security concerns from OWASP audit and CodeQL by enforcing strict
host key verification and TLS certificate verification.

Security Changes:
- Enforced strict SSH host key checking in pkg/container and devops.
- Removed insecure SSH host key verification from pkg/ansible.
- Added synchronous host key discovery during VM boot using ssh-keyscan.
- Updated UniFi client to enforce TLS certificate verification by default.
- Added --insecure flag and config option for UniFi to allow opt-in to
  skipping TLS verification for self-signed certificates.

CI and Maintenance:
- Fixed auto-merge workflow by providing repository context to 'gh' command.
- Resolved merge conflicts in .github/workflows/auto-merge.yml.
- Added unit tests for secured Ansible SSH client.
- Fixed formatting issues identified by QA checks.

* fix: gofmt alignment in cmd_config.go

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Secure connections, fix CI auto-merge, and resolve formatting

Addresses OWASP security audit and CodeQL security alerts by enforcing
secure defaults for SSH and TLS connections.

Key changes:
- Enforced strict SSH host key checking (StrictHostKeyChecking=yes).
- Implemented synchronous host key verification during VM boot using ssh-keyscan.
- Updated UniFi client to enforce TLS certificate verification by default.
- Added --insecure flag and config option for UniFi to allow opt-in to
  skipping TLS verification.
- Fixed auto-merge workflow by providing repository context to 'gh' command.
- Resolved merge conflicts in .github/workflows/auto-merge.yml.
- Fixed formatting in internal/cmd/unifi/cmd_config.go and pkg/io/local/client.go.
- Added unit tests for secured Ansible SSH client.

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Claude <developers@lethean.io>
2026-02-05 07:09:34 +00:00
Snider
7741360bd5 Migrate pkg/container to io.Medium abstraction (#292)
* chore(io): migrate pkg/container to Medium abstraction

Migrated State, Templates, and LinuxKitManager in pkg/container to use
the io.Medium abstraction for storage operations.

- Introduced TemplateManager struct to handle template logic with injected medium.
- Updated State struct to use injected medium for persistence.
- Updated LinuxKitManager to hold and use an io.Medium instance.
- Updated all internal callers in internal/cmd/vm and pkg/devops to use new APIs.
- Adapted and maintained comprehensive test coverage in linuxkit_test.go.
- Fixed naming collision with standard io package by aliasing it as goio.

* chore(io): migrate pkg/container to Medium abstraction (v2)

- Migrated State, Templates, and LinuxKitManager in pkg/container to use io.Medium.
- Introduced TemplateManager struct for dependency injection.
- Updated all call sites in internal/cmd/vm and pkg/devops.
- Restored and adapted comprehensive test suite in linuxkit_test.go.
- Fixed naming collisions and followed project test naming conventions.

* chore(io): address PR feedback for container Medium migration

- Added Open method to io.Medium interface to support log streaming.
- Implemented Open in local.Medium and MockMedium.
- Fixed extension inconsistency in GetTemplate (.yml vs .yaml).
- Refactored TemplateManager to use configurable WorkingDir and HomeDir.
- Reused TemplateManager instance in cmd_templates.go.
- Updated LinuxKitManager to use medium.Open for log access.
- Maintained and updated all tests to verify these improvements.
2026-02-04 15:33:22 +00:00
Snider
7ccfa92c7e Migrate pkg/devops to Medium abstraction (#293)
* chore(io): migrate pkg/devops to Medium abstraction

This commit migrates the pkg/devops package to use the io.Medium abstraction instead of direct calls to io.Local or the os package.

Changes:
- Updated DevOps, ImageManager, and Manifest structs to hold an io.Medium.
- Updated New, NewImageManager, and LoadConfig to accept an io.Medium.
- Updated ImageSource interface and its implementations (GitHubSource, CDNSource) to accept io.Medium in Download method.
- Refactored internal helper functions (hasFile, hasPackageScript, etc.) to use io.Medium.
- Updated all unit tests and CLI entry points to pass the appropriate io.Medium.

This migration improves the testability and flexibility of the devops package by allowing for different storage backends.

* chore(io): migrate pkg/devops to Medium abstraction

This commit completes the migration of the pkg/devops package to the io.Medium abstraction.

Changes:
- Refactored DevOps, ImageManager, and Manifest structs to use io.Medium for storage operations.
- Updated New, NewImageManager, and LoadConfig to accept an io.Medium.
- Updated ImageSource interface and its implementations (GitHubSource, CDNSource) to accept io.Medium in Download method.
- Refactored internal helper functions (hasFile, hasPackageScript, etc.) to use io.Medium.
- Updated all unit tests and CLI entry points to pass the appropriate io.Medium.
- Fixed formatting issues in test files.

This migration enables easier testing and supports alternative storage backends.
2026-02-04 14:58:03 +00:00
Snider
f2bc912ebe feat: infrastructure packages and lint cleanup (#281)
* ci: consolidate duplicate workflows and merge CodeQL configs

Remove 17 duplicate workflow files that were split copies of the
combined originals. Each family (CI, CodeQL, Coverage, PR Build,
Alpha Release) had the same job duplicated across separate
push/pull_request/schedule/manual trigger files.

Merge codeql.yml and codescan.yml into a single codeql.yml with
a language matrix covering go, javascript-typescript, python,
and actions — matching the previous default setup coverage.

Remaining workflows (one per family):
- ci.yml (push + PR + manual)
- codeql.yml (push + PR + schedule, all languages)
- coverage.yml (push + PR + manual)
- alpha-release.yml (push + manual)
- pr-build.yml (PR + manual)
- release.yml (tag push)
- agent-verify.yml, auto-label.yml, auto-project.yml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add collect, config, crypt, plugin packages and fix all lint issues

Add four new infrastructure packages with CLI commands:
- pkg/config: layered configuration (defaults → file → env → flags)
- pkg/crypt: crypto primitives (Argon2id, AES-GCM, ChaCha20, HMAC, checksums)
- pkg/plugin: plugin system with GitHub-based install/update/remove
- pkg/collect: collection subsystem (GitHub, BitcoinTalk, market, papers, excavate)

Fix all golangci-lint issues across the entire codebase (~100 errcheck,
staticcheck SA1012/SA1019/ST1005, unused, ineffassign fixes) so that
`core go qa` passes with 0 issues.

Closes #167, #168, #170, #250, #251, #252, #253, #254, #255, #256

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-04 11:34:43 +00:00
Snider
12779ef67c feat(help): add markdown parsing and section extraction (#174)
* feat(help): add markdown parsing and section extraction

Implements #137: markdown parsing and section extraction for help system.

- Add Topic and Section types for help content structure
- Add Frontmatter type for YAML metadata parsing
- Add ParseTopic() to parse markdown files into Topic structs
- Add ExtractFrontmatter() to extract YAML frontmatter
- Add ExtractSections() to extract headings and content
- Add GenerateID() to create URL-safe anchor IDs
- Add comprehensive tests following _Good/_Bad naming convention

This is the foundation for the display-agnostic help system (#133).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(test): use manual cleanup for TestDevOps_Boot_Good_FreshWithNoExisting

Fixes flaky test that fails with "TempDir RemoveAll cleanup: directory
not empty" by using os.MkdirTemp with t.Cleanup instead of t.TempDir().

This is the same fix applied to TestDevOps_Boot_Good_Success in 3423e48.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(help): address CodeRabbit review feedback

- Add CRLF line ending support to frontmatter regex
- Add empty frontmatter block support
- Use filepath.Base/Ext for cross-platform path handling
- Add tests for CRLF and empty frontmatter cases

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(help): add full-text search functionality (#175)

* fix(test): use manual cleanup for TestDevOps_Boot_Good_FreshWithNoExisting

Fixes flaky test that fails with "TempDir RemoveAll cleanup: directory
not empty" by using os.MkdirTemp with t.Cleanup instead of t.TempDir().

This is the same fix applied to TestDevOps_Boot_Good_Success in 3423e48.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(help): add full-text search functionality

Implements #139: full-text search for help topics.

- Add searchIndex with inverted index for fast lookups
- Add tokenize() for case-insensitive word extraction
- Add Search() with relevance ranking:
  - Exact word matches score 1.0
  - Prefix matches score 0.5
  - Title matches get 2.0 boost
- Add snippet extraction for search result context
- Add section-level matching for precise results
- Add comprehensive tests following _Good/_Bad naming

Search features:
- Case-insensitive matching
- Partial word matching (prefix)
- Title boost (matches in title rank higher)
- Section-level results
- Snippet extraction with context

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(help): address CodeRabbit review feedback

- Add CRLF line ending support to frontmatter regex
- Add empty frontmatter block support
- Use filepath.Base/Ext for cross-platform path handling
- Add tests for CRLF and empty frontmatter cases

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* fix(help): use rune-based slicing for UTF-8 safe snippets

Address CodeRabbit feedback: byte-based slicing can corrupt multi-byte
UTF-8 characters. Now uses rune-based indexing for snippet extraction.

- Convert content to []rune before slicing
- Convert byte position to rune position for match location
- Add UTF-8 validation tests with Japanese text

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(help): use correct string for byte-to-rune conversion in extractSnippet

strings.ToLower can change byte lengths for certain Unicode characters
(e.g., K U+212A 3 bytes → k 1 byte). Since matchPos is a byte index from
strings.Index(contentLower, word), the rune conversion must also use
contentLower to maintain correct index alignment.

Fixes CodeRabbit review feedback.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-02 00:07:32 +00:00
Snider
3423e48682 fix(test): use manual cleanup for TestDevOps_Boot_Good_Success
The test was flaky because t.TempDir() fails cleanup when files are
added asynchronously by the container manager. Using os.MkdirTemp with
manual os.RemoveAll cleanup handles this gracefully.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 11:55:10 +00:00
Snider
0072650fd9 feat: git command, build improvements, and go fmt git-aware (#74)
* feat(go): make go fmt git-aware by default

- By default, only check changed Go files (modified, staged, untracked)
- Add --all flag to check all files (previous behaviour)
- Reduces noise when running fmt on large codebases

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(build): minimal output by default, add missing i18n

- Default output now shows single line: "Success Built N artifacts (dir)"
- Add --verbose/-v flag to show full detailed output
- Add all missing i18n translations for build commands
- Errors still show failure reason in minimal mode

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add root-level `core git` command

- Create pkg/gitcmd with git workflow commands as root menu
- Export command builders from pkg/dev (AddCommitCommand, etc.)
- Commands available under both `core git` and `core dev` for compatibility
- Git commands: health, commit, push, pull, work, sync, apply
- GitHub orchestration stays in dev: issues, reviews, ci, impact

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(qa): add docblock coverage checking

Implement docblock/docstring coverage analysis for Go code:
- New `core qa docblock` command to check coverage
- Shows compact file:line list when under threshold
- Integrate with `core go qa` as a default check
- Add --docblock-threshold flag (default 80%)

The checker uses Go AST parsing to find exported symbols
(functions, types, consts, vars) without documentation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: address CodeRabbit review feedback

- Fix doc comment: "status" → "health" in gitcmd package
- Implement --check flag for `core go fmt` (exits non-zero if files need formatting)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs: add docstrings for 100% coverage

Add documentation comments to all exported symbols:
- pkg/build: ProjectType constants
- pkg/cli: LogLevel, RenderStyle, TableStyle
- pkg/framework: ServiceFor, MustServiceFor, Core.Core
- pkg/git: GitError.Error, GitError.Unwrap
- pkg/i18n: Handler Match/Handle methods
- pkg/log: Level constants
- pkg/mcp: Tool input/output types
- pkg/php: Service constants, QA types, service methods
- pkg/process: ServiceError.Error
- pkg/repos: RepoType constants
- pkg/setup: ChangeType, ChangeCategory constants
- pkg/workspace: AddWorkspaceCommands

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: standardize line endings to LF

Add .gitattributes to enforce LF line endings for all text files.
Normalize all existing files to use Unix-style line endings.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: address CodeRabbit review feedback

- cmd_format.go: validate --check/--fix mutual exclusivity, capture stderr
- cmd_docblock.go: return error instead of os.Exit(1) for proper error handling

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: address CodeRabbit review feedback (round 2)

- linuxkit.go: propagate state update errors, handle cmd.Wait() errors in waitForExit
- mcp.go: guard against empty old_string in editDiff to prevent runaway edits
- cmd_docblock.go: log parse errors instead of silently skipping

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 10:48:44 +00:00
Snider
ebc67ed727 fix(devops): fix flaky test cleanup in TestDevOps_Boot_Good_FreshFlag
Use os.MkdirTemp with explicit cleanup instead of t.TempDir() to avoid
cleanup errors when subdirectories are created during test execution.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 06:53:15 +00:00
Snider
c9ebb7c781 test: increase coverage to 63.8% across packages
Coverage improvements:
- pkg/build: 89.4%
- pkg/release: 86.7% (from 36.7%)
- pkg/container: 85.7%
- pkg/php: 62.1% (from 26%)
- pkg/devops: 56.7% (from 33.1%)
- pkg/release/publishers: 54.7%

Also:
- Add GEMINI.md for Gemini agent guidance
- Update .gitignore to exclude coverage files
- Remove stray core.go at root
- Add core go cov command for coverage reports

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-29 14:28:23 +00:00