Stackable credentials refer to the practice of accumulating a series of credentials such as certifications or microcredentials over time.  In today’s rapidly evolving job market, staying ahead of the curve is crucial for career success. Employers are increasingly looking for candidates with a diverse set of skills and qualifications. This has given rise to the concept of “stackable credentials,” a strategy that can supercharge your career trajectory. In this blog post, we’ll explore what stackable credentials are, why they matter, and how you can leverage them to open new doors in your professional journey.

What is a Stackable Credential?

University certificate credentialStackable credentials are where you can add multiple, related credentials to your resume.  Instead of pursuing a single, traditional degree – which often takes a long time, like 4 years – individuals can enhance their skill set by stacking various credentials on top of each other while they work in the field. These could include industry certifications, specialized training programs, or workshops that target specific skills.  For example, instead of getting a generalized 4 year degree in Marketing, you might take a 3 month bootcamp in WordPress and then start working in the field, while adding other credentials on search engine optimization, Google Ads, or other related topics.

Why Stackable Credentials Matter

  1. Adaptability in a Changing Job Market: The job market is dynamic, and industries are constantly evolving. Stackable credentials allow professionals to adapt to these changes quickly. By continually adding relevant certifications, individuals can stay current with industry trends and remain competitive in the job market.
  2. Customized Learning Paths: Unlike traditional degrees that follow a predetermined curriculum, stackable credentials empower individuals to tailor their learning paths. This flexibility enables professionals to focus on acquiring skills that align with their career goals, making their journey more efficient and purposeful.
  3. Incremental Career Advancement: Stackable credentials provide incremental steps for career advancement. As professionals accumulate qualifications, they can showcase a diverse skill set, making them more appealing to employers seeking candidates with a broad range of competencies. This can lead to promotions, salary increases, and expanded job responsibilities.
  4. Demonstration of Lifelong Learning: In a world where continuous learning is essential, stackable credentials demonstrate a commitment to lifelong learning. This commitment is highly valued by employers who seek individuals eager to stay informed about industry developments and emerging technologies.

Leveraging Stackable Credentials for Success

  1. Identify Your Career Goals: Begin by identifying your long-term career goals. What skills are in demand in your industry? What certifications are recognized and respected? Understanding your objectives will guide you in selecting the most relevant stackable credentials.
  2. Research Accredited Programs: It’s crucial to choose reputable and accredited programs when pursuing stackable credentials. Look for certifications and courses offered by well-known organizations or institutions. This ensures that your efforts are recognized and valued in the professional sphere.
  3. Networking Opportunities: Many stackable credential programs provide opportunities for networking with industry professionals. Take advantage of these connections to gain insights into current trends, job opportunities, and potential mentors who can guide you in your career journey.
  4. Stay Informed and Updated: Stackable credentials lose their value if they become outdated. Stay informed about changes in your industry and continuously update your skill set. Consider regularly adding new credentials to your stack to remain relevant and marketable.

External Resources for Further Learning:

  1. edX – MicroMasters Programs: Explore edX’s MicroMasters programs for in-depth, specialized learning that can be stacked for advanced qualifications.
  2. LinkedIn Learning: Access a vast library of courses on LinkedIn Learning to acquire new skills and stack credentials relevant to your career.
  3. CompTIA Certifications: Explore CompTIA’s certifications for IT professionals, offering a stackable approach to building expertise in various technology domains.
  4. Coursera Specializations: Coursera offers specializations that allow you to stack credentials in specific fields, providing a comprehensive learning path.

Summary

In conclusion, stackable credentials offer a strategic approach to career development in an ever-changing professional landscape. By embracing this mindset and continuously adding valuable qualifications, you can unlock new opportunities and elevate your career to new heights. Stay curious, stay relevant, and keep stacking those credentials for a brighter future.

value-of-digital-badges

Digital badges (aka e-badges) have emerged in today’s digitalized world as a powerful tool for recognizing and showcasing individual’s accomplishments in an online format which is more comprehensive, immediate, brandable, and easily verifiable compared to traditional paper certificates or diplomas. In this blog post, we will delve into the world of digital badges, explore best badging practices, discuss their advantages in education, and highlight examples of badge-awarding platforms.

Digital Badges and Their Utilization

Digital badges are visual representations or icons that are applied to recognize and verify an individual’s achievement or mastery of a particular skill or knowledge area in the online space. They serve as a form of digital credentialing or micro-credentialing, providing a way to display their skills, knowledge, or accomplishments in a specific domain, helping them stand out, and gain recognition in the digital landscape.

Digital badges often contain metadata, such as the name of the issuing organization, a brief description of the achievement, criteria for earning the badge, and evidence of the individual’s accomplishment. This metadata is embedded within the badge image using a technology called Open Badges, allowing anyone to verify the authenticity and details of the badge.

Digital badges are typically issued and exhibited electronically what makes them easily shareable and accessible across various digital platforms, such as social media profiles, resumes, portfolios, or online learning platforms. Digital badges are widely used in various contexts, including education, professional development, training programs, online courses, and gamification.

Digital Badging Best Practices

digital-badges-picture

To ensure the credibility of digital badges, it is important to adhere to the most effective badging practices. Here are some key practices to consider:

  • Clearly Defined Criteria: Badges should have well-defined criteria that outline the specific skills or achievements required to earn the badge. This ensures that the badge holds value and meaning.
  • Authenticity and Verification: Badges should incorporate metadata via technologies like Open Badges, enabling easy verification of the badge’s authenticity and details. This helps maintain trust and credibility in the digital badge ecosystem.
  • Issuer Credibility: Reputable organizations, institutions, or experts in the field should issue badges. The credibility of the issuer adds value to the badge and increases its recognition.
  • Visual Design: Badges should be visually appealing and distinct, making them easily recognizable and shareable. Thoughtful design elements can enhance the badge’s appeal and encourage individuals to display them proudly.

Advantages of Using Digital Badges in Education and Workforce Development

Digital badges offer numerous advantages in educational settings, transforming the way achievements and skills are recognized and valued. Below you may find some key advantages listed:

  • Granular Skill Recognition: Badges provide a way to recognize and demonstrate specific skills or achievements that might not be captured by traditional grades or degrees. This allows individuals to highlight their unique strengths and expertise.
  • Motivation and Engagement: Badges can act as a motivational tool, driving learners to actively pursue goals, and master new skills. The visual nature of badges and the ability to display them publicly create a sense of achievement and pride.
  • Portable and Shareable: Digital badges can be easily shared across various platforms, such as social media profiles, resumes, or online portfolios. This increases the visibility of accomplishments and facilitates networking and professional opportunities.
  • Lifelong Learning: Badges promote a culture of lifelong learning by recognizing and acknowledging continuous skill development. Individuals can earn badges for completing online courses, attending workshops, or acquiring new competencies, fostering a commitment to ongoing personal and professional growth.
  • Brand enhancement: Badges provide exposure to the brand of the issuing institution.
  • Turnaround time: Badges can often be available immediately on a profile page after the credential is awarded.
  • Verifiability: Digital badges are easily verifiable when using an appropriate platform/technology. This helps the consumer, hiring manager, or other stakeholder to determine if a person has the skills and knowledge that are relevant to the credential.

Examples of Badge-Awarding Platforms

Open badges platforms

Several platforms have emerged to facilitate the creation, issuance, and display of digital badges. Here are a few notable examples:

  • Credly is a widely used platform that allows organizations to design, issue, and manage digital badges. It also provides features for verifying badges and displaying them on various online platforms.
  • Open Badge Factory is an open-source platform that enables badge creation, management, and verification. It offers customizable badge templates and integration with various learning management systems.
  • Badgr is a platform that supports the creation and awarding of digital badges. It provides features for badge design, verification, and integration with various learning management systems (LMS) and online platforms.
  • BadgeCert is another platform which supports digital badges and can be integrated into other platforms.

A Certification Management System (CMS) or Credential Management System (CMS) plays a pivotal role in streamlining the key processes surrounding the certification or credentialing of people, namely that they have certain knowledge or skills in a profession.  It helps with ensuring compliance, reducing business operation costs, and maximizing the value of certifications. In this article, we explore the significance of adopting a CMS and its benefits for both nonprofits and businesses.  In today’s fast-paced and competitive business landscape, managing certifications and credentials efficiently is crucial for organizations.

What is a certification management system?

A certification management system is an enterprise software platform that is designed specifically for organizations whose primary goal is to award credentials to people for professional skills and knowledge.  Such an organization is often a nonprofit like an Association or Board, and is sometimes called an “awarding body” or similar term.   However, nowadays there are many for-profit corporations which offer certifications.  For example, many IT/Software companies will certify people on their products.  Here’s a page that does nothing but list the certifications offered by SalesForce!

These organizations often offer various credentials within a field.

  • Initial certifications (high stakes and broad) – Example: Certified Widgetmaker
  • Certificates or microcredentials – Example: Advanced Widget Design Specialist
  • Recertification exams – Example: taking a test every 5 years to maintain your Certified Widgetmaker status
  • Benchmark/progress exams – Example: Widgetmaker training programs are 2 years long and you take a benchmark exam at the end of Year 1
  • Practice tests: Example: old items from the Certified Widgetmaker test provided in a low-stakes fashion for practice

A credentialing body will need to manage the many business and operation aspects around these.  Some examples:

  • Applications, tracking who is applying for whichCertification management system, credentialing
  • Payment processing
  • Eligibility pathways and documentation (e.g., diplomas)
  • Pass/Fail results
  • Retake status
  • Expiration dates

There will often be functionality that makes these things easier, like automated emails to remind the professionals when their certification is expiring so they can register for their Recertification exam.

 

Reasons to use a certification management system

  1. Enhancing Compliance and Regulatory Adherence: In industries with stringent compliance requirements, such as healthcare, finance, and IT, adhering to regulations and maintaining accurate records of certifications is paramount. A comprehensive CMS provides a centralized repository where organizations can securely store, track, and manage certifications and credentials. This ensures compliance with industry standards, regulatory bodies, and audits. With automated alerts and renewal notifications, organizations can stay on top of certification expirations, reducing the risk of non-compliance and associated penalties.  A certification management system will also help your organization achieve accreditation like NCCA or ANSI/ISO 17024.
  2. Streamlining Certification Tracking and Renewals: Managing certifications manually can be a time-consuming and error-prone process. A CMS simplifies this task by automating certification tracking, renewal reminders, and verification processes. By digitizing the management of certifications, organizations can save valuable time and resources, eliminating the need for tedious paperwork and manual record-keeping. Additionally, employees can easily access their certification status, track progress, and initiate renewal processes through a user-friendly interface, enhancing transparency and self-service capabilities.
  3. Improving Workforce Efficiency and Development: An efficient CMS empowers organizations to optimize their workforce’s knowledge and skill development. By capturing comprehensive data on certifications, skills, and training, organizations gain valuable insights into their employees’ capabilities. This information can guide targeted training initiatives, succession planning, and talent management efforts. Moreover, employees can leverage the CMS to identify skill gaps, explore potential career paths, and pursue professional development opportunities. This aligns individual aspirations with organizational goals, fostering a culture of continuous learning and growth.
  4. Enhancing Credential Verification and Fraud Prevention: Verifying the authenticity of certifications is critical, especially in industries where credentials hold significant weight. A CMS with built-in verification features enables employers, clients, and other stakeholders to authenticate certifications quickly and accurately. By incorporating advanced security measures, such as blockchain technology or encrypted digital badges, CMSs provide an added layer of protection against fraud and credential forgery. This not only safeguards the reputation of organizations but also fosters trust and confidence among customers, partners, and regulatory bodies.

 

Of course, the bottom line is that a certification management system will save money, because this is a lot of information for the awarding body to track, and it is mission-critical.

 

Conclusion

Implementing a Certification Management System or Credential Management System is a strategic investment for organizations seeking to streamline their certification processes and maximize their value. By centralizing certification management, enhancing compliance, streamlining renewals, improving workforce development, and bolstering credential verification, a robust CMS empowers organizations to stay ahead in a competitive landscape while ensuring credibility and regulatory adherence.

ANSI ISO/IEC 17024 accreditation is an internationally recognized standard for the accreditation of personnel certification bodies.  That is, it is a stamp of approval from an independent audit which says your certification is good quality.  ANSI stands for the American National Standards Institute, while ISO refers to the International Organization for Standardization. The portion of ANSI which carries out the accreditation process is the ANSI National Accreditation Board (ANAB).

What does ANSI ISO/IEC 17024 cover?

ANSI ISO/IEC 17024 specifies the requirements for bodies operating certification programs for individuals, ensuring that the certification processes are fair, valid, and reliable. The standard outlines the general principles and requirements for the certification of personnel across various fields, including but not limited to healthcare, information technology, engineering, and safety.

The standard covers a wide range of aspects related to certification bodies, including:

  1. Impartiality and independence: Certification bodies must demonstrate impartiality and avoid any ccertification accreditationonflicts of interest.
  2. Certification program development: The standard sets criteria for developing certification programs, including defining competencies, establishing eligibility requirements, and developing examination processes.
  3. Examination processes: It outlines guidelines for the design, development, and administration of examinations to assess individuals’ knowledge, skills, and competencies.
  4. Certification process: The standard addresses the application process, evaluation of candidates, decision-making on certification, and ongoing certification maintenance.
  5. Management system requirements: ANSI ISO/IEC 17024 includes requirements for the management system of the certification body, including document control, record keeping, and continual improvement processes.

What does ANSI ISO/IEC 17024 mean?

Accreditation to ANSI ISO/IEC 17024 provides assurance to stakeholders that the certification programs and processes are conducted in a consistent, competent, and reliable manner. It enhances the credibility and acceptance of certifications issued by accredited certification bodies, helping individuals demonstrate their professional competence and expertise in their respective fields.

Should my organization pursue accreditation?

That is a business question for you.  In some cases, you are required; in some professions, there might be a law that candidates do not receive federal funding or do not have certifications recognized if their certification is not accredited.  However, for many professions, accreditation is optional.  In those cases, if there are two certification bodies, it is a competitive advantage for one to become accredited.  But for small certification bodies with no competitors, accreditation is often not worth the great expense.

Here are some reasons to consider pursuing accreditation.

  1. Global Recognition: ISO 17024 accreditation provides global recognition and credibility to a certification program. It demonstrates compliance with internationally recognized standards, which can enhance the reputation of the certification and increase its acceptance worldwide.
  2. Quality Assurance: ISO 17024 accreditation ensures that the certification program follows rigorous processes and standards for the development, administration, and evaluation of assessments. This helps maintain the quality and reliability of the certification, giving stakeholders confidence in its validity and fairness.
  3. Competitive Advantage: Accreditation under ISO 17024 can serve as a competitive differentiator for the certification program. It distinguishes the certification from others in the market by indicating a commitment to high standards of professionalism, competence, and integrity.
  4. Stakeholder Confidence: Accreditation provides assurance to stakeholders, including employers, professionals, and regulatory bodies, that the certification meets recognized criteria for competency assessment. This builds trust and confidence in the certification, leading to increased participation and recognition within the industry.
  5. Continuous Improvement: ISO 17024 accreditation fosters a culture of continuous improvement within the certification program. Through regular assessments and audits, organizations can identify areas for enhancement and implement best practices to enhance the effectiveness and relevance of the certification over time.

Note that ANSI ISO/IEC 17024 is not the only show in town.  The National Commission for Certifying Agencies also accredits certifications, though they define it per certification program, not certification body.

Do I need to do all this work myself?

No!  Much of it, yes, you need to do, because no one else has the specific knowledge of your profession and content area.  But we can certainly help with some portions, especially the exam development and psychometrics.  We can also provide the item banking and exam delivery platform to securely administer your exams and report the results.

Microcredentials are short, focused, and targeted educational or assessment-based certificate programs that offer learners a way to acquire specific skills or knowledge in a particular field.  In today’s fast-paced and rapidly evolving job market, traditional degrees may not always be enough to stand out among the competition, and they often take too long to achieve. Microcredentials have emerged as a promising solution to this problem.

What is a Microcredential?

A microcredential is, as the name would suggest, a smaller version of a credential.  A credential is an attestation of learning or skills, that is typically long-term or large scale.  Examples are a 4-year degree in Marketing, certification as a SalesForce developer, or licensure as a surgeon.  Microcredentials have a similar meaning but are typically much more narrow; instead of a degree in marketing, you might have completed various course/certificates in WordPress plugin development, Google Ads, Search Engine Optimization, etc.

In addition, there are other terms like “nano-degrees” or “digital badges” which sometimes overlap, but there are no agreed-upon definitions that differentiate.  However, Badges are usually considered to be even smaller than a Microcredential or Nano-Degree.

In most cases, microcredentials are tied to educational programs, and are therefore quite distinct from certification or licensure, which are assessment-focused.  These are often smaller than a university degree, such as a 2-week online course or a 3-month bootcamp.

Why Microcredentials?

Microcredentials have become increasingly popular in recent years because they offer several benefits to both learners and employers. For learners, they provimicrocredentials-degree-onlinede a more flexible and cost-effective way to gain new skills or upgrade existing ones without committing to a full-time degree program.  Additionally, microcredentials allow learners to demonstrate their competency in a specific skill or knowledge area, which can help them stand out in a crowded job market.  Given that they are less expensive and short duration, learners can often receive “more bang for their buck” in terms of adding to their skillset and improving job prospects.

For employers, microcredentials offer a way to identify job candidates who possess the specific skills they need. By looking for candidates who have earned they in relevant areas, employers can quickly narrow down the pool of applicants and find the most qualified candidates. Additionally, they can be used by employers to upskill their existing workforce, helping employees stay current with the latest developments in their field.

How to Earn Microcredentials

Microcredentials can be earned in a variety of ways, including online courses, workshops, boot camps, and other short-term training programs. These programs typically focus on a specific topic, such as data analytics, project management, or digital marketing. To earn a microcredential, learners must complete a series of assessments or projects that demonstrate their mastery of the subject matter. Once earned, microcredentials are typically displayed as digital badges that can be shared on social media profiles, online resumes, or other platforms.

Types of Microcredentials

There are several types of microcredentials available, including skill-based, competency-based, and stackable credentials. Skill-based microcredentials focus on developing specific skills or knowledge areas, such as coding, graphic design, or language proficiency. Competency-based microcredentials, on the other hand, assess a learner’s ability to perform a specific task or set of tasks, such as managing a team or conducting market research. Finally, stackable microcredentials allow learners to build on existing credentials by earning additional microcredentials in related areas, creating a pathway to a full degree program.

Examples

Consider the field of marketing.  Traditionally, you might go to a university for a 4-year degree in Marketing, Business, or Communications.  This is a very broad approach, and of course takes 4 years (or more for some people).  Alternatively, there are now many options where you can get a microcredential focused specifically on Digital Marketing, a very in-demand skill set.  A great example of this is Oregon State University, but a quick googling will show you many more.  Some providers will even get more specific, such as Social Media Management, Search Engine Optimization, or even specifically WordPress.  But then these are typically branded as a Badge rather than a Nano-Degree or Microcredential.

Conclusion

In conclusion, microcredentials offer a flexible and cost-effective way to gain new skills or upgrade existing ones, making them an attractive option for both learners and employers. By focusing on specific skills or knowledge areas, they allow learners to demonstrate their competence in a particular field, while also providing a way for employers to identify the most qualified job candidates. As the job market continues to evolve, microcredentials are likely to become an increasingly important part of the educational landscape, providing learners with the tools they need to succeed in their careers.

The Ebel method of standard setting is a psychometric approach to establish a cutscore for tests consisting of multiple-choice questions. It is usually used for high-stakes examinations in the fields of higher education, medical and health professions, and for selecting applicants.

How is the Ebel method performed?

The Ebel method requires a panel of judges who would first categorize each item in a data set by two criteria: level of difficulty and relevance or importance. Then the panel would agree upon an expected percentage of items that should be answered correctly for each group of items according to their categorization.

It is crucial that judges are the experts in the examined field; otherwise, their judgement would not be valid and reliable. Prior to the item rating process, the panelists should be given sufficient amount of information about the purpose and procedures of the Ebel method. In particular, it is important that the judges would understand the meaning of difficulty and relevance in the context of the current assessment.

Next stage would be to determine what “minimally competent” performance means in the specific case depending on the content. When everything is clear and all definitions are agreed upon, the experts should classify each item across difficulty (easy, medium, or hard) and relevance (minimal, acceptable, important, or essential). In order to minimize the influence of the judges’ opinion on each other, it is more recommended to use individual ratings rather than consensus ones.

Afterwards judgements on the proportion of items expected to be answered correctly by minimally competent candidates need to be collected for each item category, e.g. easy and desirable. However, for the rating and timesaving purposes the grid proposed by Ebel and Frisbie (1972) might be used. It is worth mentioning though that Ebel ratings are content-specific, so values in the grid might happen to be too low or too high for a test.

Ebel-method-data

At the end, the Ebel method, like the modified-Angoff method, identifies a cut-off score for an examination based on the performance of candidates in relation to a defined standard (absolute), rather than how they perform in relation to their peers (relative). Ebel scores for each item and for the whole exam are calculated as the average of the scores provided by each expert: the number of items in each category is multiplied by the expected percentage of correct answers, and the total results are added to calculate the cutscore.

Pros of using Ebel

  • This method provides an overview of a test difficulty
  • Cut-off score is identified prior to an examination
  • It is relatively easy for experts to perform

Cons of using Ebel

  • This method is time-consuming and costly
  • Evaluation grid is hard to get right
  • Digital software is required
  • Back-up is necessary

Conclusion

The Ebel method is a quite complex standard-setting process compared to others due to the need of an analysis of the content, and it therefore imposes a burden on the standard-setting panel. However, Ebel considers the relevance of the test items and the expected proportion of the correct answers of the minimally competent candidates, including borderline candidates. Thus, even though the procedure is complicated, the results are very stable and very close to the actual cut-off scores.

References

Ebel, R. L., & Frisbie, D. A. (1972). Essentials of educational measurement.

Certification exam delivery is the process of administering a certification test to candidates.  This might seem straightforward, but it is surprisingly complex.  The greater the scale and the stakes, the more potential threats and pitfalls.

For starters, you need to make sure the exam was developed well in the first place.  Learn more about that here.

1. Determine Approach to Certification Exam Delivery

The first step is to determine how you want to deliver the certification exam.  Here are your options for exam delivery.

  • Test Centers
    • Your locations
    • Third party
  • Mode of delivery
    • Paper
    • Online
      • Unproctored
      • Remotely Proctored (Recorded)
      • Remotely Proctored (Live)
      • Multi-modal (a mix of the above)

Which approach to use for certification exam delivery depends on several factors.

  • Are you going to seek accreditation?
  • Do you have an easy way to make your own locations?  One example is that you have quarterly regional conferences for your profession, where you could simply get a side room to deliver your test to candidates since they will already be there.  Another is that most of your candidates are coming from training programs at universities, and you are able to use classrooms at those universities.
  • What are the stakes of the exam?  In rare cases, it can be unproctored online.  Low- to medium-stakes can be delivered with recorded video proctoring.  The highest stakes exams will almost always be at test centers with in-person proctoring, including strong measures like fingerprinting and retinal scanning.  Learn more about remote proctoring.
  • How many exams are you planning to deliver?  If a large volume, higher-cost approaches might be out of your budget.
  • What is your testing cycle like?  Some exams are on-demand year-round, while some might be one or two Saturdays per year.

2. Shop for Vendors

Once you have some idea what you are looking for, start shopping for vendors that provide services for certification exam delivery, development, and scoring.  In some cases, you might not settle on a certain approach right away, and that’s OK.  See what is out there and compare prices.  Perhaps the cost of Live Remote Proctoring is more affordable than you anticipated, and you can upgrade to that.

Besides a simple Google search some good places to start are the member listings of the Association of Test Publishers and the Institute for Credentialing Excellence.

3. Create Policies and Documentation for Certification Exam Delivery

Once you have finalized your vendors, you need to write policies and documentation around them.  For example, if your vendor has a certain login page for proctoring (we have ascproctor.com), you should take relevant screenshots and write up a walkthrough so candidates know what to expect.  Much of this should go into your Candidate Handbook.  Some of the things to cover:

exam development certification committee

  • How to prepare for the exam
  • How to take a practice test
  • What is allowed during the exam
  • What is not allowed
  • ID needed and the check-in process
  • Details on specific locations (if using locations)
  • Time limits and other practical considerations in the exam

4. Let Everyone Know

Once you have written up everything, make sure all the relevant stakeholders know.  Publish the new Candidate Handbook and announce to the world.  Send emails to all upcoming candidates with instructions and an opportunity for a practice exam.  Put a link on your homepage.  Get in touch with all the training programs or universities in your field.  Make sure that everyone has ample opportunity to know about the new process!

5. Roll Out

Finally, of course, you can implement the new approach to certification exam delivery.  You might launch a new certification exam from scratch, or perhaps you are moving one from paper to online with remote proctoring, or some other change.  Either way, you need a date to start using it and a change management process.  The good news is that, even though it’s probably a lot of work to get here, the new approach is probably going to save you time and money in the long run.  Roll it out!


Contact Us To Speak to a Psychometrician

 

A cutscore or passing point (aka cut-off score and cutoff score as well) is a score on a test that is used to categorized examinees.  The most common example of this is pass/fail, which we are all familiar with from our school days.  For instance, a score of 70% and above will pass, while below 70% will fail.  However, many tests have more than one cutscore.  An example of this is the National Assessment of Educational Progress (NAEP) in the USA, which as 3 cutscores, creating 4 categories: Below Basic, Basic, Proficient, and Advanced.

The process of setting a cutscore is called a standard-setting study.  However, I dislike this term because the word “standard” is used to reflect other things in the assessment world.  In some cases, it is the definition of what is to be learned or covered (see Common Core State Standards) and in other cases it refers to the process of reducing construct-irrelevant variance by ensuring that all examinees are taking the testing in standardized conditions (standardized testing).  So I prefer cutscore or passing point.  And passing point is limited to the case of an exam with only one cutscore where the classifications are pass/fail, which is not always the case – not only are there many situations where there are more than one cutscore, but many two-category situations might use other decisions, like Hire/NotHire or a clinical diagnosis like Depressed/NotDepressed.

Types of cutscores

There are two types of cutscores, reflecting the two ways that a test score can be interpreted: Norm-referenced and criterion-referenced.

Criterion-referenced Cutscore

A cutscore of this type is referenced to the material of the exam, regardless of examinee performance.  In most cases, this is the sort of cutscore that you need to be legally defensible for high stakes exams.  Psychometricians have spent a lot of time inventing ways to do this, and scientifically studying them.

Names of some methods you might see for this type are: modified-Angoff, Nedelsky, and Bookmark.

Example

An example of this is a certification exam.  If the cutscore is 75%, you pass.  In some months or years, this might be most candidates, in other months it might be fewer.  The standard does not change.  In fact, the organizations that manage such exams go to great lengths to keep it stable over time, a process known as equating.

Norm-referenced Cutscore

A cutscore of this type is referenced to the examinees, regardless of their mastery of the material.

A name of this you might see is a quota.  Such as when a test is delivered to only accept the top 10% of applicants.

Example

An example of this was in my college Biology class.  It was a weeder class, to weed out the students who start college planning to be pre-med simply because they like the idea of being a doctor or are drawn to the potential salary.  So, the exams were intentionally made very hard, so that the average score might only be 50% correct.  They then awarded an A to anyone who had a z-score of 1.0 or greater, which is the top 15% of students – regardless of how well you actually scored on the exam.  You might get a score of 60% correct but be 95th percentile and get an A.

The Nedelsky method is an approach to setting the cutscore of an exam.  Originally suggested by Nedelsky (1954), it is an early attempt to implement a quantitative, rigorous procedure to the process of standard setting.  Quantitative approaches are needed to eliminate the arbitrariness and subjectivity that would otherwise dominate the process of setting a cutscore.  The most obvious and common example of this is simply setting the cutscore at a round number like 70%, regardless of the difficulty of the test or the ability level of the examinees.  It is for this reason that a cutscore must be set with a method such as the Nedelsky approach to be legally defensible or meet accreditation standards.

How to implement the Nedelsky method

The first step, like several other standard setting methods, is to gather a panel of subject matter experts (SMEs).  The next step is for the panel to discuss the concept of a minimally qualified candidate   This is a concept about the type of candidate that should barely pass this exam, and sits on the borderline of competence. They then review a test form, paying specific attention to each of the items on the form.  For every item in the test form, each rater estimates the number of options that an MCC will be able to eliminate.  This then translates into the probability of a correct response, assuming that each candidate guesses amongst the remaining options.   If an MCC can only eliminate one of the options of a four option item, they then have a 1/3 = 33% chance of getting the item correct.  If two, then ½ = 50%.

These ratings are then averaged across all items and all raters.  This then represents the percentage score expected of an MCC on this test form, as defined by the panel.  This makes a compelling, quantitative argument for what the cutscore should then be, because we would expect anyone that is minimally qualified to score at that point or higher.

Item Rater1 Rater2 Rater3
1 33 50 33
2 25 25 25
3 25 33 25
4 33 50 50
5 50 100 50
Total 33.2 51.6 36.6

Drawbacks to the Nedelsky method

This approach only works on multiple choice items, because it depends on the evaluation of option probability.  It is also a gross oversimplification.  If the item has four options, there are only four possible values for the Nedelsky rating 25%, 33%, 50%, 100%.  This is all the more striking when you consider that most items tend to have a percent-correct value between 50% and 100%, and reflecting this fact is impossible with the Nedelsky method. Obviously, more goes into answering a question than simply eliminating one or two of the distractors.  This is one reason that another method is generally preferred and supersedes this method…

Nedelsky vs Modified-Angoff

The Nedelsky method has been superseded by the modified-Angoff method.  The modified-Angoff method is essentially the same process but allows for finer variations, and can be applied to other item types.  The modified-Angoff method subsumes the Nedelsky method, as a rater can still implement the Nedelsky approach within that paradigm.  In fact, I often tell raters to use the Nedelsky approach as a starting point or benchmark.  For example, if they think that the examinee can easily eliminate two options, and is slightly more likely to guess one of the remaining two options, the rating is not 50%, but rather 60%.  The modified-Angoff approach also allows for a second round of ratings after discussion to increase consensus (Delphi Method).  Raters can slightly adjust their rating without being hemmed into one of only four possible ratings.

An Objective Structured Clinical Examination (OSCE Exam) is an assessment designed to measure performance of tasks, typically medical, in a high-fidelity way.  It is more a test of skill than knowledge.  For example, I used to work at a certification board for ophthalmic assistants; there were 3 levels, and the top two levels included both a knowledge test (200 multiple choice items) and an OSCE (level 2 was a digital simulation, level 3 was live human patients).

OSCE exams serve a very important purpose in many fields, forging a critical bridge between learning and practice.  This post will cover some of the basics.

What is an Objective Structured Clinical Examination?

An OSCE exam typically works by defining very specific tasks that the examinee is required to do, while examiners (often professors) watch them while grading them via a rubric or checklist.  Each of the tasks is often called a station, and the OSCE will often have multiple stations.  Consider the components of the name:

  • Objective: We are trying to be as objective as possible, boiling down a potentially very complex patient scenario and task into a checklist or rubric. We want to make it quantitative, measurable, and reliable.
  • Structured: The task itself is very boxed, such as using retinoscopy to measure astigmatism (perhaps one thing of 20 that might happen at a visit to your ophthalmologist).
  • Clinical: The task is something to be done in a clinical setting; this is to increase fidelity and validity.

A great summary is provided by Zayyan (2011):

The Objective Structured Clinical Examination is a versatile multipurpose evaluative tool that can be utilized to assess health care professionals in a clinical setting. It assesses competency, based on objective testing through direct observation. It is precise, objective, and reproducible allowing uniform testing ofclinical examination students for a wide range of clinical skills. Unlike the traditional clinical exam, the OSCE could evaluate areas most critical to performance of health care professionals such as communication skills and ability to handle unpredictable patient behavior.

There are a few key points here.

  • It is a clinical setting, rather than a lecture hall setting (though in non-medical fields, “clinical setting” is relative!)
  • It is assessing competency of clinical skills
  • It is based on observation, where examiners rate the examinee
  • It will often include assessment of “soft skills” or other non-knowledge aspects

Where are OSCE Exams used?

OSCE exams are very important in the medical professions.  This report shows that many medical schools use it, though it curiously does not say how many schools were part of the survey.

However, it is most certainly not limited to medical fields.  You don’t hear the term very often outside medical education, but the approach is used widely.   Professions where someone is physically doing something are more likely to use OSCEs.  An accountant, on the other hand, does no physically do something, and their equivalent of an OSCE is more like a complex accounting scenario that needs to be completed in MS Excel and then graded.

Examples of OSCE exams

Of course, there are many medical examples.  I work with the American Board of Chiropractic Sports Physicians, who have a practical exam.  Check out their DACBSP webpage and scroll down to the Practical Exam resources, including instructional videos for some stations.

Nurse skill test

I once worked with a crane operator certification.  They had a performance test where you had to drive the crane into a certain position, lift and place certain objects, and then move a wrecking ball through a path of oil drums without knocking anything over – all while being rated by an examiner with a checklist.  Sounds a lot like an OSCE?

Perhaps the most common OSCE is one that you have likely taken: a Driver’s test.  In addition to taking a knowledge test, you were also likely asked to drive a car with an examiner armed with a checklist while he told you to do various “stations” like parallel parking, perpendicular parking, or navigating a stoplight.

Tell me more!

There are dedicated resources in the world of medical education and assessment, such as Downing and Yudkowsky (2019) Assessment in Health Professions Education (https://www.routledge.com/Assessment-in-Health-Professions-Education/Yudkowsky-Park-Downing/p/book/9781315166902).   You might also be interested in my Lecture Notes from a course taught using that textbook.