Assessing Skill Levels of IT Auditors

Assessing Skill Levels of IT Auditors

The job role of “Cybersecurity” is a silly term. Why? Because NIST states that there are actually 52 different work roles and 33 specialist areas in Cybersecurity. So, in some places, it’s a bit like asking advertising for a GP for a brain surgery post.

Introduction

Auditing is such an important aspect of our working lives. While many will dread the call of the auditor; it does provide a way to make sure that things are done correctly. And, so, for cybersecurity, we increasingly need auditors who can get into the details of a system and understand how code and systems interact. This is especially important with the move to the public cloud, and where you just can’t point to physical systems. But, some in the industry see auditing as a tick-box activity:

Has encryption? Yes
Does it log events? Yes
Are digital certificates updated? Yes
Do employees have a CCNA certification? Yes

A recent PhD thesis by Blake Curtis on “Creating the Next Generation Cybersecurity Auditor: Examining the Relationship between IT Auditors’ Competency, Audit Quality, & Data Breaches” [here][1] provides an insight into whether auditors lack hands-on experience in their theoretical and practical understanding.

For this, the work analysed knowledge of areas for 108 IT auditors and 108 subject matter experts (SMEs) such as related to AWS, Microsoft Servers, firewalls, Kubernetes containers, and Microsoft Azure. This included declarative knowledge (“theory”) and procedural knowledge (“practice”) questions. Overall, the research includes most of the key auditing companies, such as Deloitte, PwC, Ernst & Young, and KPMG.

NIST NICE

NIST has tried to simplify the complexity of cybersecurity, and for all the various stakeholders involved:

But our current mappings between industry and academia do not quite match up the things that industry would define within job roles, where academia often fails to link with professional development.

At one time an academic qualification might be the thing that gets you your promotion, but increasingly it's more about the knowledge and skills that you have and your career progression. NIST thus have defined standardized names for roles, knowledge, skills and abilities, and which will help recruiters to identify the best candidates and for companies to plan their skills development activities.

NIST have thus broken the cybersecurity into categories and then split these into specialty areas, work roles, skills, knowledge areas and abilities. There are 33 specialist areas, and these map to seven categories: Securely Provision (SP); Operate and Maintain (OM); Oversee and Govern (OV); Protect and Defend (PR); Analyze (AN); Collect and Operate (CO); and Investigate (IN):

We then have seven categories; 33 specialty areas; and 52 work roles [here], and these then map these to 1,007 tasks, 374 skills, 630 knowledge areas and 176 abilities:

• Securely Provision (SP). Risk Management (RSK); Software Development (DEV); Systems Architecture (ARC); Technology R&D (TRD); Systems Requirements Planning (SRP); Test and Evaluation (TST)
• Operate and Maintain (OM). Data Administration (DTA). Knowledge Management (KMG). Customer Service and Technical Support (STS); Network Services (NET); Systems Administration (ADM); Systems Analysis (ANA)
• Oversee and Govern (OV). Legal Advice and Advocacy (LGA); Training, Education, and Awareness (TEA); Cybersecurity Management (MGT); Strategic Planning and Policy (SPP); Executive Cyber Leadership (EXL); Program/Project Management (PMA) and Acquisition;
• Protect and Defend (PR). Cybersecurity Defense Analysis (CDA); Cybersecurity Defense Infrastructure Support (INF); Incident Response (CIR); Vulnerability Assessment and Management (VAM)
• Analyze (AN). Threat Analysis (TWA); Exploitation Analysis (EXP); All-Source Analysis (ASA); Targets (TGT); Language Analysis (LNG).
• Collect and Operate (CO). Collection Operations (CLO); Cyber Operational Planning (OPL); Cyber Operations (OPS).
• Investigate (IN). Cyber Investigation (INV); Digital Forensics (FOR).

Auditing Roles: IT Program Auditor and Security Control Assessor

Within the NIST-defined NICE (National Initiative for Cybersecurity Education) framework, we have the work role of OV-PMA-005 ( IT Program Auditor) and SP-RSK-002 (Security Control Assessor). OV-PMA-005 is included within the OVERSEE and GOVERN (OV) specialty area (leadership, management, direction, or development and advocacy so the organization may effectively conduct cybersecurity work), and SP-RSK-002 is included within the SECURITY PROVISION specialty area (Conceptualizes, designs, procures, and/or builds secure information technology (IT) systems, with responsibility for aspects of system and/or network development).

Key findings

The thesis conducted an evaluation of both the theoretical knowledge of the key skills based on the NICE framework, along with their practical skills. The assessment levels include: follow, assist, apply, enable, and ensure:

The Follow level is the most basic, and Ensure is the highest level of application and knowledge. The scoring used the SFIA (Skills Framework for the Information Age) approach. For knowledge, the average knowledge score of IT auditors was at Level 3 (Apply), but Subject Experts came in at Level 4 (Ensure):

For procedural knowledge, the research analysed practical (“hands-on”) knowledge. For this, IT Auditors came in at the Assist level, where Subject Experts achieved the Apply level:

The research matched the declarative and procedural knowledge to recent data breaches (such as the Capital One breach in 2019), and where a lack of knowledge in key areas could lead to an increased risk level for an organisation.

Overall, the research found that auditors — on average — matched to the “Apply” level for procedural knowledge, but only to “Assist” in applying this knowledge. For SMEs, again there was a decline from “Enable” for theoretical knowledge to “Apply” for practical knowledge.

Certifications make you confident?

In opposition to this, generally, IT auditors had high levels of self-perception confidence in converting technical evidence into an overall conclusion, but this was weaker for SMEs. Another finding was that age had an effect on confidence levels, and that confidence increases with age. But, the study found, that experience levels did not generally boost performance levels.

The two certifications which produced high levels of confidence for IT Auditors were Certified Information System Auditor (CISA) and Certified Information Systems Security Professional (CISSP). Those that had these generally felt Very Confident, but only had an overall score of “Apply”. As with SMEs who were Very Confident, the overall average score was “Apply”:

Thus, it may be concluded that certification increases confidence levels, but does not nescarily boost knowledge to the same extent. When it came to the degree level (BSc, MSc and PhD), the findings found that there was no real correlation with performance levels.

One other finding was the vendor-specific, cybersecurity and IT auditing training with hands-on experience generally improved performance.

Conclusions

The main findings of this work is that there is generally sufficient declarative knowledge (book knowledge) for IT auditors and subject matter experts (SMEs) have sufficient levels of declarative knowledge (book knowledge), but insufficient procedural knowledge (the application of knowledge) for IT auditors. For this, the research identifies that the current levels of training and certification of auditors do not include enough task-based skills assessments.

There are 33 expert areas within cybersecurity, and it is folly to think that many auditors can cover many of the areas. This can be likened to medicine, where we have GPs and specialists. While a GP has general knowledge, a specialist is often required for specific knowledge.

Overall, perhaps, we need to have stronger support for specialisations in auditing and make sure these specialists have hands-on experience.

If you are interested in the work roles, try here:

And to test your knowledge of work roles, try this test:

https://asecuritysite.com/tests/fun?sortBy=nice

References

[1] Curtis, B. (2022). Creating the Next Generation Cybersecurity Auditor: Examining the Relationship between It Auditors’ Competency, Audit Quality, & Data Breaches (Doctoral dissertation, Capitol Technology University). [here]