Principal AI Engineer for Autonomous Security-Testing Agents

Role Summary

Lead the design, training, evaluation, and deployment of production-grade, on-premise multi-agent LLM systems for autonomous security testing. You own the full stack: self-hosted open-weight models (fine-tuned when it helps), the agent orchestration that drives them, the GPU infrastructure that serves them, and the safety and governance around them. This role is on-site in Bandung, in secure and air-gapped environments, on hardware we operate ourselves.

 We move fast. You should be able to take a rough idea and have something running the same day, then harden it once it proves out. The stack, the models, and the threats all change quickly, and we expect you to adapt with them rather than wait for a finished spec.

Key Responsibilities

• Lead LLM development end to end for security-testing tasks: dataset curation,SFT/LoRA/QLoRA, DPO/RLHF, and large-scale automated evaluation against held-out benchmarks that measure genuine capability, not memorized answers.
• Design and own multi-agent orchestration: goal decomposition, tool use via  function calling, structured outputs, verifier-gated outputs, fallbacks, retries, and recovery when a long-running task stalls or corrupts state.
• Build and harden the agent's tool layer: integrate security tooling behind a stable, well-typed interface, and manage the retrieval layer that grounds     the agent's decisions.
• Run the red-team and safety harness: simulate and execute authorized adversarial techniques (injection, traversal, prompt injection, data exfiltration) against in-scope targets, and build the guards, intensity controls, and policy gates that keep runs scoped, auditable, and safe.
• Conduct adversarial and robustness testing for NLP/CV models: distribution shift, perturbations, poisoning risk, and the mitigations for them.
• Own performance and cost on self-operated hardware: quantization (GGUF, GPTQ, AWQ, FP8, int4), tensor parallelism, KV-cache management, chunked prefill, batching, speculative decoding, and GPU utilization.
• Operate the GPU fleet: deploy, swap, and version models across multiple GPU classes and multi-node setups, handling parallelism, NCCL, memory limits, and secure network routing for air-gapped sites.
• Architect retrieval and grounding: vector stores, chunking, embeddings, hybrid  search, re-ranking, and retrieval-quality evaluation.
• Develop production APIs and services (FastAPI, gRPC, WebSocket) with observability, live run tracing, canarying, and human-in-the-loop review. Monitor quality and safety drift, and handle incidents.
• Keep work reproducible and governed: experiment tracking, model registry, artifact stores, versioned configs, run-log retention, and clean benchmark  hygiene with no train/eval contamination.
• Deploy and operate on-prem (VMs, Docker, Kubernetes) with versioning,rollback, autoscaling, and secure upgrade paths for air-gapped, regulatedsites.
• Collaborate and mentor: scope experiments with the team, write design docs,  threat models, and runbooks, drive code reviews, and grow junior engineers.

 Requirements

• Bachelor's or Master's in CS, AI, or a related field, or equivalent  experience.
• 5+ years in applied ML/AI and 10+ years in software engineering.
• Strong Python and hands-on PyTorch (and/or TensorFlow).
• LLM fine-tuning experience: SFT, LoRA/QLoRA, DPO or RLHF, plus dataset prep,  synthetic data generation, and large-scale evaluation.
• Self-hosted inference experience with at least one open-weight family (Llama,Qwen, Mistral, Gemma) on vLLM, SGLang, TGI, or llama.cpp, including quantization and tensor parallelism.
• Multi-agent design and tool-use orchestration: function calling, tool integration, structured outputs, error handling, retries, and verifier/guard patterns.
• RAG pipelines with a vector store (pgvector, Milvus, Weaviate), plus embedding selection and retrieval-quality evaluation.
• MLOps: Docker, Kubernetes, Git, CI/CD, experiment tracking, model registry, and data/version management.
• Production monitoring and observability: logging, tracing, metrics, safety evaluation, SLOs, and alerting.
• Security and safety practice: prompt-injection defenses, PII handling, RBAC, secrets management, and audit logging, with comfort in regulated or air-gapped environments.
• A bias toward shipping: you prototype quickly, iterate in a fast-changing environment, and harden what works without waiting for perfect requirements.
• Willing to work on-site in Bandung.

Strongly Preferred

• Offensive-security, penetration-testing, or CTF experience: you've solved, or built systems that solve, real exploitation tasks. Adversarial ML and robustness background a plus.
• Secure model deployment in government or critical-infrastructure contexts.
• Deep transformer, tokenization, and quantization internals.
• GPU systems expertise: CUDA, NCCL, MIG, multi-node deployment, and performance  profiling, including current-generation accelerators.
• Hugging Face ecosystem and optimized-inference toolchains.