What Makes a Personality Test Accurate?

In 1948, psychologist Bertram Forer handed his students a personality analysis. He told them it was personalized, built from their own test answers when, in fact, every student received the same report, assembled from a newspaper astrology column. Each analysis had about as much scientific backing as a “What Garbage Matches Your Personality?” quiz on Buzzfeed. Yet, when asked to rate how accurately it described them, the class awarded their analysis an overall 4.26 out of 5, a result that has been repeated with almost identical scores in the decades since. 

The Forer effect, as it has since become known, describes a psychological phenomenon where people believe that vague and generalized personality descriptions apply specifically and uniquely to them. Even though the descriptions could fit almost anyone, people rate them as very accurate. And the more flattering the description, the more likely someone is to believe it is true. 

Why this happens is complex. We humans have a desire for meaning and try hard to connect general statements to our own lives. We also have a positivity bias and easily accept flattering traits as true for ourselves. And of course, psychologists have proven our deep-seated trust in and obedience to authority through several landmark experiments (Milgram shock experiment; Bickman’s uniform study). These studies suggest that people will believe the results of any assessment if a supposed expert gives them.

Put all of this together, and we see that most people will evaluate the accuracy of a personality test, especially an expert-backed test, by asking whether the result feels right, a subjective feeling that can be manufactured with no measurement behind it at all. This is problematic for business leaders. If an organization trusts a test solely because it "feels true," how will it know if the test is actually measuring something real they can safely base team-building decisions on, rather than an expensive piece of fortune-cookie pseudoscience? 

The Qualities of an Accurate Personality Test

If resonance isn't proof of accuracy, what is? 

The answer to that question lives entirely outside the result a test-taker sees, and entirely inside a body of research completed long before the first question is written. That research covers how a publisher defines what it is measuring, how rigorously its items were built and tested, how large and representative a sample stands behind its norms, and how thoroughly its scores have been checked against real behavior. 

The more expansive this research, the more distance a test puts between itself and the astrology column Forer borrowed from. A credible test will fight the bias through several key mechanisms.

1. Does the Test Define What It's Actually Measuring?

Before any statistic gets involved, a credible test must name a specific, objective, defined trait for every score it produces (instead of vague, universally flattering statements). Psychologists call this a construct. Examples include honesty, fear, grit, perfectionism and friendliness. 

Since personality constructs are invisible, they must be defined precisely enough that a researcher can write a question targeting exactly that trait. For example, extraversion and similar adjectival traits like “outgoingness” and "assertiveness" are internal qualities that we cannot see directly. We study them by looking at how people act and answer questions like “I like talking to strangers” or “Being the center of attention is a good thing.”

All established personality frameworks have clearly defined constructs that have been tested over time. DISC, for example, has the constructs of Drive, Influence, Support and Clarity. The Big Five has five constructs known by the acronym OCEAN: Openness, Conscientiousness, Extraversion, Agreeableness and Neuroticism. Myers-Briggs is built on four dichotomous preference pairs (constructs) that combine to create 16 distinct personality types: Extraversion (E) vs. Introversion (I), Sensing (S) vs. Intuition (N), Thinking (T) vs. Feeling (F) and Judging (J) vs. Perceiving (P).  These constructs weren't settled overnight—they've been refined, debated and re-tested across decades of research, giving each one a body of evidence most brand-new proprietary scales simply haven't had time to accumulate. 

If you’re using a proprietary test that isn’t built on an established system, one that has been created from scratch, it’s a good idea to check what constructs are being assessed and how they have been fleshed out. That work should have happened long before the questions were compiled. If a publisher cannot name the exact trait a scale measures, no later validation study can be trusted to mean what it claims.

2. Is the Test Reliable? 

A scientifically sound test must yield stable results over time. Personality traits do not change drastically from week to week so if you take the test today, and take it again in six weeks under the same conditions, your score should be nearly identical. We call this reliability, and it is measured using statistical methods like test-retest reliability.  A test-retest coefficient of 0.70 or higher is generally considered acceptable, with 0.80 and above indicating strong reliability.

A second reliability benchmark is internal consistency, meaning that the questions within the test agree with each other when measuring the same trait with no contradictions. For example, if a test is measuring construct "extraversion," your answers to a question about enjoying crowded parties should statistically align with your answers about striking up conversations with strangers. Look for a metric called Cronbach’s alpha (α) in the technical documentation. A reputable psychological assessment will have an alpha value of 0.70 or higher for each construct it measures.

It’s worth mentioning that typology systems like the Enneagram and Myers-Briggs face ongoing criticism from mainstream psychologists regarding their relatively poor test-retest reliability. This is usually a criticism of type theory itself, as personality traits often exist on fluid spectrums (as more widely accepted scientific models like the Big Five personality traits and the Holland Code measure) than the distinct boxes these type systems place people into. Plus, variability in performance isn’t unusual for self-report assessments since mood, context and self-perception can shift someone's answers slightly from one sitting to the next, even when the underlying trait hasn't changed at all.  

That said, for every study that demonstrates poor reliability, there’s another that demonstrates strong reliability, especially for Myers-Briggs which has 80 years of testing and application behind it. Organizations can find reassurance inside the publisher’s technical documentation. For example, Truity publishes technical documentation for each of our assessments, citing our extremely large dataset testing (millions of users) and standard reliability metrics across each construct. If that documentation doesn't exist or isn't available on request, treat the test as unproven, no matter how polished the publisher’s marketing looks. 

3. Does the Test Predict Real-world Outcomes?

Validity analysis is complicated and several checks take place before a credible test gets published. Together, they ensure that the test measures what it claims to measure, checked against real outcomes outside the test itself. 

Breaking that down, some checks verify that the traits a test claims to measure are the ones that actually emerge from the data.  So if a test creator builds 10 questions to measure Extraversion and 10 questions to measure Agreeableness on the Big Five, so-called factorial validity is achieved when people's answers prove that those two groups of questions stay completely separate, rather than mixing together into a confusing mess.

Other methods assess the degree to which a test score accurately forecasts a person's future behavior, performance or real-world outcomes. For example, if a team member scores high for Conscientiousness, predictive validity is proven if that person actually turns out to be a high-performing, organized, dependable employee six months later. Again, you should find the publisher’s analysis in its technical documentation.

Some personality frameworks have greater predictive power than others. The Big Five, in particular, has the strongest predictive validity in the academic literature, with decades of studies linking each factor to concrete outcomes like job performance, relationship satisfaction and even longevity.

Typology-based systems like Myers-Briggs have a more mixed track record, largely because forcing people into one of two boxes on each dichotomy discards important information about how strongly someone leans one way or the other. Two people may type as Introverted, for example, but placing a person who is 51% Introverted into the same "I" category as someone who is 99% Introverted creates categories that do not actually exist in nature.

However, researcher John A. Johnson has pointed out that the four Myers-Briggs scales correlate meaningfully with four of the Big Five factors (E-I with Extraversion, S-N with Openness, T-F with Agreeableness, and J-P with Conscientiousness), suggesting the underlying traits themselves have real predictive value even if the classic Myers-Briggs format doesn't capture it well.  His proposed fix is to abandon the strict either-or dichotomy and instead report someone's results on a continuous scale, preserving the nuance that gets lost when a person is simply labeled "E" or "I." 

Some newer assessments built on Myers and Briggs' original theory, including Truity's own TypeFinder personality assessment, have adopted this scale-based approach.

4. Does The Test Account for People Answering to Look Good?

Most popular personality assessments are self-reported, and with that comes the risk that respondents game the system by answering in a way that reflects well on them, especially if a performance review or their reputation depends on the result. Researchers call this “social desirability” bias. If a test doesn’t account for it, you will end up with artificially inflated scores for positive traits while negative, relevant traits stay hidden. 

To minimize social desirability bias, a credible test will write questions in a way that avoids implausibly virtuous answer patterns. For example, it will frame questions neutrally, avoiding loaded words that imply a "correct" moral choice or a socially preferred answer and/or will provide balanced multiple-choice response options that offer an equal number of positive and negative choices. Some tests include a ‘forced-choice’ format where respondents are asked to rank statements against each. A test maker who knows what they’re doing will group statements (in pairs, triplets or quads) that carry the same level of social appeal or social unattractiveness so respondents are forced into a genuine trade-off and cannot simply pick the most flattering option. 

A leader using test results in any high-stakes setting should ask specifically how the publisher addresses this. A test with no answer to that question will produce inflated results.

5. Is the Score Backed by a Large, Relevant Sample?

Many personality tests use normative scoring, where a respondent's answers are compared against a reference group to produce a percentile or trait score that can be compared across people. Normative scoring evaluates a test-taker’s performance relative to others—it shows who possesses more or less of a trait compared to peers. 

Sample size matters greatly here. A norm group of 100 university students will, quite obviously, produce a very different picture of “high”, “low” and "average" than one drawn from tens of thousands of working adults across industries, ages and countries. A credible publisher will build norm samples at genuine scale, often in the millions, and they will either keep that sample demographically broad or customized by industry (e.g., tech, healthcare) if that is more relevant to the specific population being assessed.

Some assessments use ipsative scoring instead of normative scoring. This is a forced-choice format that compares a person's traits only to their other traits, resulting in a self-referenced profile of relative strengths and weaknesses. Ipsative scoring drastically reduces the social desirability bias we talked about in the previous section by forcing trade-offs between equally attractive statements, but the drawback is you cannot mathematically compare Person A's score to Person B's score. Two people could have the exact same profile but vastly different absolute levels of those traits.

No method is objectively better than the other, and some tests are ipsative in their core philosophy. Myers-Briggs, for example, forces you to choose between two opposing preferences (e.g., Introversion vs. Extraversion), so it obviously compares the test-taker’s internal preferences against each other to arrive at a four-letter type (like INFJ), rather than a percentile rank against the general public. The Big Five is more normative, and other frameworks use hybrid adaptive testing, combining a whole bunch of question formats and testing to build highly accurate personality profiles. 

Ultimately, this comes down to transparency: a credible publisher will tell you plainly which scoring method they use and why, along with the size and makeup of any norm group behind it. If that information is hard to find or vaguely worded, it's a signal the publisher may not have done the underlying work, and that's a much bigger red flag for accuracy than what type of scoring is used.  

Putting It All Together

This five-point 'accuracy' checklist is the difference between a tool that supports the coaching conversations, development plans and team decisions organizations actually use it for, and one that just feels true. The Forer effect works precisely because vague flattery is easy to believe; rigor in the test’s design is what stops flattery from being mistaken for insight.

So if you’re evaluating a test with no data behind it, understand you may be purchasing Forer's horoscope. To have faith in a test's accuracy, you need to review the technical manual and see what's happening behind the scenes. You don't have to bore your employees with statistics, but you do need to know the numbers exist, because you're the one who has to be able to defend the result. And when those results feed into formal HR or compliance processes, that documentation is what you'll be asked for.