Automated Penetration Testing for Government Compliance: How JST Cut Testing Cycles from Years to Days

The Japan Science and Technology Agency (JST) is one of Japan's most important national institutions. As a government body responsible for funding R&D projects, conducting policy-driven research, and promoting public understanding of science and technology, JST sits at the center of Japan's innovation ecosystem - and at the center of a significant cybersecurity challenge.

With 1,546 employees and hundreds of critical information systems supporting national research programs, JST operates under Japan's Unified Cybersecurity Standards for Government Agencies. As a major R&D funding body, it is also a high-value target for sophisticated cyber threats. The combination of regulatory compliance pressure and active threat exposure makes security testing not just a compliance exercise, but an operational necessity.

The problem was scale. JST had been conducting vulnerability assessments and penetration tests manually - even before unified standards formally required them. But with hundreds of systems across diverse business operations, running tests sequentially meant that completing a full cycle for all systems often took two to three years. By the time a system was retested, its threat landscape had changed entirely.

As Tsutomu Kurosawa of JST put it: "Security controls are essential to protect JST's mission, but our goal is also to maximize research efficiency. We needed a solution that balances both."

ULTRA RED was selected following a formal government bidding process in late 2023 to solve exactly this problem.

The Challenge: Hundreds of Systems, a Two-to-Three Year Testing Backlog

JST's security testing challenge had four compounding dimensions:

• Testing cycles stretched to 2-3 years: With hundreds of systems tested sequentially by manual teams, individual systems were assessed infrequently. A vulnerability that appeared after a system's last test would remain undetected until the next cycle - which could be years away.

• Government compliance requirements that evolve continuously: Japan's Unified Cybersecurity Standards for Government Agencies are updated regularly to reflect emerging attack trends. Keeping pace with standards that change faster than a two-year testing cycle is structurally impossible with manual approaches.

• High-value target status: As a major R&D funding body, JST is an attractive target for advanced persistent threats and state-sponsored actors interested in research data, funding allocations, and innovation pipelines. The threat level is not comparable to a typical enterprise environment.

• Growing external attack surface: Cloud adoption and the expansion of external-facing assets created additional blind spots beyond what manual testing programs had been scoped to cover.

Why Manual Testing Could Not Scale

Manual penetration testing has real value - skilled testers find complex logic flaws and attack paths that automated tools miss. But the economics of manual testing break down at JST's scale. With hundreds of systems, sequential manual cycles create a fundamental gap: the time between tests is measured in years, not months.

For a government agency operating under continuously updated cybersecurity standards, a finding discovered in year one of a three-year cycle that is not remediated until year three represents years of unaddressed exposure. And any new system deployed, any cloud service adopted, or any configuration change made between cycles falls entirely outside the testing scope.

This is the core problem that continuous, automated penetration testing solves. Rather than testing each system once every few years, automated platforms run assessments continuously - weekly, or even daily - across all systems simultaneously. The compliance evidence trail stays current. New exposures are detected as they appear. And the security posture reflects the actual current state of the environment, not a snapshot from two years ago.

How JST Evaluated and Selected ULTRA RED

In November 2023, JST conducted a formal proof of concept, evaluating solutions against four specific requirements:

• Automating penetration testing and exposure validation across all monitored systems

• Simulating real-world attacker techniques at high frequency - not just signature-based scanning

• Integrating the latest threat intelligence to detect emerging risks as they appear

• Scaling across hundreds of systems with minimal operational burden on JST's security team

After a comprehensive evaluation of proposals from multiple vendors through a formal government bidding process, ULTRA RED was selected. As Kurosawa noted: "ULTRA RED met our key requirements and provided a unified platform, robust detection and automation capabilities."

ULTRA RED was chosen specifically for its ability to deliver a Continuous Threat Exposure Management (CTEM) program through an agentless SaaS platform that integrates three capabilities in a single platform: External Attack Surface Management (EASM) for continuous discovery and monitoring of exposed IT assets; Automated Breach and Attack Simulation (ABAS) for realistic penetration tests based on the latest attack techniques; and Cyber Threat Intelligence (CTI) that continuously updates testing techniques to reflect emerging attacker behaviors.

Deployment followed in July 2024.

What Automated Testing Looks Like in a Government Environment

JST's implementation reflects what a mature automated security testing program looks like for a government agency operating at scale:

Testing frequency moved from a 2-3 year manual cycle to weekly automated intrusion simulations, with capability to scale toward daily testing. Results feed directly into JST's existing SOC, CSIRT, and security workflows. Dashboards provide real-time visibility into vulnerabilities and threat exposures. And findings are shared directly with system owners - not routed through a central queue - for rapid remediation.

Satoshi Yanagida, Systems Lead at JST, described the operational impact: "ULTRA RED highlights vulnerabilities, ranks them by business risk, and facilitates collaboration between security and system owners."

This last point matters for government environments specifically. Security teams in government agencies often operate at a distance from the system owners responsible for individual applications. Platforms that route findings directly to owners - with business risk context and remediation guidance - remove a coordination bottleneck that slows remediation in traditional security programs.

JST successfully passed an external audit following the ULTRA RED deployment. The full case study is available in the ULTRA RED success stories.

Automated vs. Manual Penetration Testing for Government Compliance

For government agencies and regulated organizations, the right approach is not automated or manual - it is both, applied at the right layer. See automated vs. manual penetration testing for a full comparison.

Manual penetration testing delivers depth on high-value, complex systems - particularly for custom applications, logic flaws, and sophisticated attack path discovery. It remains essential for priority systems and formal compliance engagements that specifically require human testers.

Automated continuous testing delivers the coverage and frequency that government compliance standards increasingly require. It ensures that every system in scope is assessed continuously, not once every few years, and that the compliance evidence trail reflects current state rather than a historical snapshot.

Before and After: What Automation Changed for JST

The operational shift from manual to automated testing at JST produced measurable changes across four dimensions:

• Testing cycle: From 2-3 years per complete cycle to weekly automated simulations, with capability to scale to daily testing.

• Risk prioritization: From no systematic prioritization to automatic ranking of vulnerabilities by business impact.

• Attack coverage: From limited, point-in-time coverage to continuously updated attack scenarios reflecting the latest threat intelligence.

• Operational cost: From high projected cost of expanding manual testing to cover hundreds of systems, to lower cost with higher coverage - without expanding the security team. As Kurosawa noted, replicating ULTRA RED's testing coverage manually would have required enormous resources.

Looking Ahead: VITA AI for Further Automation

JST is exploring further use of ULTRA RED's VITA AI assistant - an AI-powered security capability designed to accelerate detection and classification of threats, automate analysis and response prioritization, and further reduce mean time to remediation (MTTR).

For a government agency managing hundreds of systems with a team of under 1,600 people total, the ability to automate not just testing but also triage and prioritization is the next step toward a security program that scales with the threat environment rather than with headcount.

Frequently Asked Questions

What cybersecurity standards apply to Japanese government agencies?

Japanese government agencies operate under the 'Unified Cybersecurity Standards for Government Agencies,' maintained and updated by Japan's National center of Incident readiness and Strategy for Cybersecurity (NISC). These standards are continuously revised to reflect emerging attack trends - which is why point-in-time compliance approaches that test each system every few years are structurally insufficient.

How does automated penetration testing satisfy government compliance requirements?

Automated penetration testing provides continuous, timestamped evidence of security assessment across all systems in scope. Rather than a point-in-time report that goes stale within months, automated platforms generate a live compliance evidence trail that reflects the current state of the environment - which is what continuously updated government cybersecurity standards expect.

What is the difference between vulnerability scanning and automated penetration testing?

Vulnerability scanning identifies potential weaknesses based on known signatures and version checks. Automated penetration testing - particularly platforms that include breach and attack simulation - actively tests whether those weaknesses are exploitable under real-world attack conditions. Government compliance frameworks are increasingly expecting proof of exploitability, not just a scan report.

How does ULTRA RED support both EASM and penetration testing in one platform?

ULTRA RED's platform integrates External Attack Surface Management (continuous discovery and monitoring of exposed assets), Automated Breach and Attack Simulation (realistic penetration tests based on current attack techniques), and Cyber Threat Intelligence (continuously updated attack scenarios). This unified approach means organizations like JST can run a single platform for both discovery and testing rather than managing separate tools for each function.

Why do government agencies need more frequent penetration testing than annual cycles?

Government agencies are high-value targets for advanced persistent threats and state-sponsored actors. Their systems change continuously through cloud adoption, new applications, and configuration updates. And the compliance standards they operate under are revised regularly. A testing cycle measured in years cannot keep pace with any of these realities - which is why frameworks like Japan's unified government standards are pushing toward continuous, automated assessment.

See how ULTRA RED supports government and enterprise security compliance programs. Book a demo or explore the platform.

Related: What Is Continuous Penetration Testing? | Automated vs. Manual Penetration Testing | What Is Continuous Threat Exposure Management? | Proof of Exploitability