<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[Carlisle Rainey]]></title><description><![CDATA[Musings about quantitative social science.]]></description><link>https://posts.carlislerainey.com</link><image><url>https://substackcdn.com/image/fetch/$s_!fRiu!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7271b474-7009-423a-a74f-8a77abf0247d_1024x1024.png</url><title>Carlisle Rainey</title><link>https://posts.carlislerainey.com</link></image><generator>Substack</generator><lastBuildDate>Sun, 22 Nov 2026 06:03:26 GMT</lastBuildDate><atom:link href="https://posts.carlislerainey.com/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Carlisle Rainey]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[carlislerainey@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[carlislerainey@substack.com]]></itunes:email><itunes:name><![CDATA[Carlisle Rainey]]></itunes:name></itunes:owner><itunes:author><![CDATA[Carlisle Rainey]]></itunes:author><googleplay:owner><![CDATA[carlislerainey@substack.com]]></googleplay:owner><googleplay:email><![CDATA[carlislerainey@substack.com]]></googleplay:email><googleplay:author><![CDATA[Carlisle Rainey]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[On AI for Quantitative Social Science]]></title><description><![CDATA[some initial, early thoughts]]></description><link>https://posts.carlislerainey.com/p/on-ai-for-quantitative-social-science</link><guid isPermaLink="false">https://posts.carlislerainey.com/p/on-ai-for-quantitative-social-science</guid><dc:creator><![CDATA[Carlisle Rainey]]></dc:creator><pubDate>Mon, 16 Nov 2026 14:03:44 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/8458054c-4d07-4fea-b3db-2fb999a7b139_1536x1024.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>The emergence of agentic AI<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a> has radically changed how we can conduct quantitative social science.<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a> It seems clear that we <em>should</em> use agentic AI in our work&#8212;critics can no longer dispute the benefits.</p><p>But how should we think about how AI models can help? How can we make sure that we do not <em>underuse</em> or <em>overuse</em> the models? I cannot answer these questions confidently. However, I can describe how I am <em>thinking about</em> these questions.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://posts.carlislerainey.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><h2>What We Do (or Perhaps &#8220;What We Did&#8221;)</h2><p>In social science, we write papers to help other social scientists understand the world. For <em>quantitative</em> social science, I think of projects as having three parts:</p><ol><li><p>A <em>dataset</em>.</p></li><li><p>A <em>paper</em>.</p></li><li><p>A <em>process</em> that takes the researcher from the dataset to the paper.</p></li></ol><p>I&#8217;ll focus on this <em>process</em> in this essay.</p><p>At our best, the process consists of well-documented computer code that implements our principled decisions (that we carefully defend in the paper).</p><p>The process is <em>long</em>. A quick look across several replication archives in political science suggests that a typical project uses one to six scripts and 500 to 2,500 lines of code. Some use many more scripts and many more lines of code.</p><p>The process is <em>important</em>. We might even argue that the <em>process</em> is the primary research product. <a href="https://web.stanford.edu/dept/statistics/cgi-bin/donoho/wp-content/uploads/2018/08/wavelab.pdf">Buckheit and Donoho (1995, 5)</a> write:</p><blockquote><p>An article about computational science in a scientific publication is <em>not</em> the scholarship itself, it is merely <em>advertising</em> of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.</p></blockquote><p>So what should this process look like&#8212;what features should it have? I suggest three:</p><ol><li><p><em>Principled</em>: the researcher made reasonable decisions in analyzing their data.</p></li><li><p><em>Implemented</em>: the researcher actually implemented those decisions.</p></li><li><p><em>Documented</em>: the researcher made it easy to check that they actually implemented those decisions.</p></li></ol><p>We could imagine, for example, that a researcher comes up with a clever research design but fails to code an important variable correctly. Or perhaps they did everything correctly, but they pointed-and-clicked their way to their final estimates and left no way to verify they did anything at all.</p><p>I&#8217;ve long advocated for what <a href="https://plain-text.co">Healy (2020)</a> calls &#8220;plain-text social science&#8221; because tools like CSV datasets, R scripts, and LaTeX files make it easier to document your process (and thus easier to implement). As <a href="https://plain-text.co">Healy (2020, 8)</a> notes:</p><blockquote><p>[S]oftware applications are not all created equal, and some make it easier than others to do the Right Thing.</p></blockquote><p>This leaves us with two uncontroversial claims:</p><ol><li><p>We should make smart choices, implement those choices, and document those choices.</p></li><li><p>We should use tools that make it easy to do those things.</p></li></ol><h2>Enter AI</h2><p>AI models enter almost seamlessly into this framework, especially when used agentically.</p><p>It is indisputable that AI models are tools that can help <em>implement</em> our choices and <em>document</em> those choices. They can write clear and correct R scripts. They can refactor and debug human-written R scripts. They can create careful records. They can document revisions.<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-3" href="#footnote-3" target="_self">3</a></p><p>This is clear: <em>at a minimum</em>, we should use AI models <em>extensively</em> for error-checking (<a href="https://pmaronow.github.io/llm-disclosure.pdf">Aronow 2026</a>).</p><h2>Mistakes We Can Make</h2><p>We can make two types of mistakes:</p><ol><li><p>We can use AI too little.</p></li><li><p>We can use AI too much (or for the wrong things).</p></li></ol><p>And for PhD students and junior faculty, it is important that they thread this needle.</p><h3>Too Little</h3><p>The risks of too little seem clear: we fail to discover and correct errors that AI models would have detected.<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-4" href="#footnote-4" target="_self">4</a></p><p><strong>Real examples of errors.</strong></p><ul><li><p><em>Example 1:</em> A paper describes a prior distribution as &#8220;N(0, 2).&#8221; From context, &#8220;2&#8221; refers to a variance. The Stan model uses <code>normal(0, 2)</code>, and Stan treats <code>2</code> as a standard deviation.</p></li><li><p><em>Example 2:</em> A paper describes a complex algorithm implemented in C++. A subtle but important difference exists between the algorithm described in the paper and the algorithm implemented by the C++ code.</p></li></ul><p>While these errors are nearly impossible for a human to detect, an AI model will almost always discover them.</p><h3>Too Much</h3><p>The risks of too much seem less clear. But here is my thinking.</p><p>First, we might turn over tasks to AI models in which the human should remain deeply involved. I suspect that current models are not great at things like justifying the substantive credibility of assumptions used for causal inference. I suspect they are not great at finding open and important questions.<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-5" href="#footnote-5" target="_self">5</a></p><p>Second, we might use AI models to quickly complete tasks that ought not be completed at all. Thus, rather than making less work, they make <em>more</em> work. It is clear that we can use agents to do lots of tasks very quickly and with many fewer errors. What matters now is that we point those agents towards <em>worthwhile</em> tasks.<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-6" href="#footnote-6" target="_self">6</a></p><h2>An Attitude for the Future</h2><p>Some researchers seem optimistic that agents will allow us to work faster, collect more data, or push theories further. Other researchers seem worried about being overwhelmed by slop. My feelings are different.</p><p>I&#8217;m excited about this: AI agents will allow us to work <em>more carefully</em>.<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-7" href="#footnote-7" target="_self">7</a></p><p>And I&#8217;m worried about this: AI agents will allow us to pursue all sorts of computational projects that are not worth doing at all.</p><p>So here&#8217;s an attitude for the future: let&#8217;s continue our work. Let&#8217;s continue to do work that&#8217;s worth doing. Let&#8217;s continue to write papers worth reading. And let&#8217;s continue to use tools that help us do that work well (even if it slows us down a little).</p><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-1" href="#footnote-anchor-1" class="footnote-number" contenteditable="false" target="_self">1</a><div class="footnote-content"><p>Such as Anthropic&#8217;s Claude Code interface and Opus 5.5 model, as of October 2026.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-2" href="#footnote-anchor-2" class="footnote-number" contenteditable="false" target="_self">2</a><div class="footnote-content"><p>Perhaps I should write &#8220;computational&#8221; social science for this essay. That&#8217;s what I mean&#8212;social science that combines quantitative datasets, data wrangling, and statistical modeling.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-3" href="#footnote-anchor-3" class="footnote-number" contenteditable="false" target="_self">3</a><div class="footnote-content"><p>They can also help make principled decisions, but work in this dimension is more complicated, I think.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-4" href="#footnote-anchor-4" class="footnote-number" contenteditable="false" target="_self">4</a><div class="footnote-content"><p>Others seem to think that AI allows us to work faster, incorporate more data, or push further theoretically. Perhaps there is lots of opportunity there, as well.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-5" href="#footnote-anchor-5" class="footnote-number" contenteditable="false" target="_self">5</a><div class="footnote-content"><p>I don&#8217;t worry about researchers overusing models for these things, though. I&#8217;m actually optimistic that researchers care about the quality of the papers they put out under their name. I suspect we have (or will quickly develop) good ways to filter out &#8220;slop.&#8221;</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-6" href="#footnote-anchor-6" class="footnote-number" contenteditable="false" target="_self">6</a><div class="footnote-content"><p>Take the example of data collection. Prior to the emergence of agents, extensive data collection was difficult, expensive, and time-consuming. The high costs made us think carefully about what we wanted and why before we went ahead. The difficulty encouraged and rewarded discernment and judgment. With AI agents, extensive data collection is easy and cheap&#8212;you can do it while you sleep. The low costs make it easy to &#8220;just go ahead&#8221; before we&#8217;ve put any thought into the matter.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-7" href="#footnote-anchor-7" class="footnote-number" contenteditable="false" target="_self">7</a><div class="footnote-content"><p>When I&#8217;ve used them best, I feel like agents have <em>slowed me down</em>, not sped me up, without a huge jump in theoretical or empirical ambition.</p></div></div>]]></content:encoded></item><item><title><![CDATA[Logistic Regression in Small (and Not So Small) Samples]]></title><description><![CDATA[&#9888;&#65039; Warning: Memes, charts, #rstats, and practical advice ahead.]]></description><link>https://posts.carlislerainey.com/p/memes-about-i</link><guid isPermaLink="false">https://posts.carlislerainey.com/p/memes-about-i</guid><dc:creator><![CDATA[Carlisle Rainey]]></dc:creator><pubDate>Mon, 09 Nov 2026 14:39:00 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!Z6xR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c5a1fa1-fbde-4eb4-a44f-71c318f5c483_500x756.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em>This post originally appeared as a thread on Twitter. You can find it <a href="https://x.com/carlislerainey/status/1686389777225113601?a=PWM5-xbIGRnzRp9mHTE&amp;s=20">here</a>. It&#8217;s a fun summary of <a href="https://www.cambridge.org/core/journals/political-science-research-and-methods/article/estimating-logit-models-with-small-samples/712F6039372475B650B123E9A1FE60E6">Rainey and McCaskey (2021)</a>.</em></p><div><hr></div><p>Do you use logistic regression? If so, you&#8217;ll want to read the thread below.</p><p>&#9888;&#65039; Warning: Memes, charts, #rstats, and practical advice ahead.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!j_Mh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37d23336-6df4-4ca0-b9c4-50a51e24c9ff_500x500.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!j_Mh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37d23336-6df4-4ca0-b9c4-50a51e24c9ff_500x500.jpeg 424w, https://substackcdn.com/image/fetch/$s_!j_Mh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37d23336-6df4-4ca0-b9c4-50a51e24c9ff_500x500.jpeg 848w, https://substackcdn.com/image/fetch/$s_!j_Mh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37d23336-6df4-4ca0-b9c4-50a51e24c9ff_500x500.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!j_Mh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37d23336-6df4-4ca0-b9c4-50a51e24c9ff_500x500.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!j_Mh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37d23336-6df4-4ca0-b9c4-50a51e24c9ff_500x500.jpeg" width="500" height="500" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/37d23336-6df4-4ca0-b9c4-50a51e24c9ff_500x500.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:500,&quot;width&quot;:500,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Meme: Anakin and Padm&#233;. Anakin: \&quot;Logit models don't work well in small samples.\&quot; Padm&#233;: \&quot;With N = 1,000, asymptotic results apply, right?\&quot; Then, unsmiling: \&quot;Right?\&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Meme: Anakin and Padm&#233;. Anakin: &quot;Logit models don't work well in small samples.&quot; Padm&#233;: &quot;With N = 1,000, asymptotic results apply, right?&quot; Then, unsmiling: &quot;Right?&quot;" title="Meme: Anakin and Padm&#233;. Anakin: &quot;Logit models don't work well in small samples.&quot; Padm&#233;: &quot;With N = 1,000, asymptotic results apply, right?&quot; Then, unsmiling: &quot;Right?&quot;" srcset="https://substackcdn.com/image/fetch/$s_!j_Mh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37d23336-6df4-4ca0-b9c4-50a51e24c9ff_500x500.jpeg 424w, https://substackcdn.com/image/fetch/$s_!j_Mh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37d23336-6df4-4ca0-b9c4-50a51e24c9ff_500x500.jpeg 848w, https://substackcdn.com/image/fetch/$s_!j_Mh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37d23336-6df4-4ca0-b9c4-50a51e24c9ff_500x500.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!j_Mh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37d23336-6df4-4ca0-b9c4-50a51e24c9ff_500x500.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>If you don&#8217;t like Twitter shenanigans, I&#8217;ll give it away.</p><p>For logit models with small-to-moderate samples (maybe N &lt; 1,000), you should consider Firth&#8217;s penalized estimator.</p><p>I talk about it in <a href="https://www.cambridge.org/core/journals/political-science-research-and-methods/article/estimating-logit-models-with-small-samples/712F6039372475B650B123E9A1FE60E6">this paper</a> with Kelly McCaskey (open access!).</p><div><hr></div><p>Now on with the thread!</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!B7U-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b4b9c6d-54e4-4232-afeb-a74ac36a8fc0_622x477.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!B7U-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b4b9c6d-54e4-4232-afeb-a74ac36a8fc0_622x477.jpeg 424w, https://substackcdn.com/image/fetch/$s_!B7U-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b4b9c6d-54e4-4232-afeb-a74ac36a8fc0_622x477.jpeg 848w, https://substackcdn.com/image/fetch/$s_!B7U-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b4b9c6d-54e4-4232-afeb-a74ac36a8fc0_622x477.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!B7U-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b4b9c6d-54e4-4232-afeb-a74ac36a8fc0_622x477.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!B7U-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b4b9c6d-54e4-4232-afeb-a74ac36a8fc0_622x477.jpeg" width="622" height="477" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3b4b9c6d-54e4-4232-afeb-a74ac36a8fc0_622x477.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:477,&quot;width&quot;:622,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Meme: Ned Stark, \&quot;Brace yourselves. A long thread on logit models with lots of memes and plots is coming.\&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Meme: Ned Stark, &quot;Brace yourselves. A long thread on logit models with lots of memes and plots is coming.&quot;" title="Meme: Ned Stark, &quot;Brace yourselves. A long thread on logit models with lots of memes and plots is coming.&quot;" srcset="https://substackcdn.com/image/fetch/$s_!B7U-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b4b9c6d-54e4-4232-afeb-a74ac36a8fc0_622x477.jpeg 424w, https://substackcdn.com/image/fetch/$s_!B7U-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b4b9c6d-54e4-4232-afeb-a74ac36a8fc0_622x477.jpeg 848w, https://substackcdn.com/image/fetch/$s_!B7U-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b4b9c6d-54e4-4232-afeb-a74ac36a8fc0_622x477.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!B7U-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b4b9c6d-54e4-4232-afeb-a74ac36a8fc0_622x477.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>Logit models are really common in the social sciences. We typically use maximum likelihood (ML) to estimate these models. But the excellent properties of these models are mostly asymptotic.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!edW4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f1f49f3-48b3-43c0-ab1c-09d866c10ce3_690x362.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!edW4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f1f49f3-48b3-43c0-ab1c-09d866c10ce3_690x362.jpeg 424w, https://substackcdn.com/image/fetch/$s_!edW4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f1f49f3-48b3-43c0-ab1c-09d866c10ce3_690x362.jpeg 848w, https://substackcdn.com/image/fetch/$s_!edW4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f1f49f3-48b3-43c0-ab1c-09d866c10ce3_690x362.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!edW4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f1f49f3-48b3-43c0-ab1c-09d866c10ce3_690x362.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!edW4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f1f49f3-48b3-43c0-ab1c-09d866c10ce3_690x362.jpeg" width="690" height="362" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3f1f49f3-48b3-43c0-ab1c-09d866c10ce3_690x362.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:362,&quot;width&quot;:690,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Meme: the \&quot;Is this a pigeon?\&quot; anime man, labeled \&quot;Social scientist,\&quot; gestures at a butterfly labeled \&quot;Asymptotic result.\&quot; Caption: \&quot;Hello neighbor.\&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Meme: the &quot;Is this a pigeon?&quot; anime man, labeled &quot;Social scientist,&quot; gestures at a butterfly labeled &quot;Asymptotic result.&quot; Caption: &quot;Hello neighbor.&quot;" title="Meme: the &quot;Is this a pigeon?&quot; anime man, labeled &quot;Social scientist,&quot; gestures at a butterfly labeled &quot;Asymptotic result.&quot; Caption: &quot;Hello neighbor.&quot;" srcset="https://substackcdn.com/image/fetch/$s_!edW4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f1f49f3-48b3-43c0-ab1c-09d866c10ce3_690x362.jpeg 424w, https://substackcdn.com/image/fetch/$s_!edW4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f1f49f3-48b3-43c0-ab1c-09d866c10ce3_690x362.jpeg 848w, https://substackcdn.com/image/fetch/$s_!edW4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f1f49f3-48b3-43c0-ab1c-09d866c10ce3_690x362.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!edW4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f1f49f3-48b3-43c0-ab1c-09d866c10ce3_690x362.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>However, these estimates might not be well-behaved in small samples. In particular, some folks are concerned about small sample bias in logit models. And that&#8217;s a real thing.</p><p>(But I don&#8217;t think it&#8217;s the most important problem&#8212;keep reading.)</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!v31g!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dbd33dd-3f25-44ee-9bba-6e015bec2dc0_1536x1154.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!v31g!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dbd33dd-3f25-44ee-9bba-6e015bec2dc0_1536x1154.png 424w, https://substackcdn.com/image/fetch/$s_!v31g!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dbd33dd-3f25-44ee-9bba-6e015bec2dc0_1536x1154.png 848w, https://substackcdn.com/image/fetch/$s_!v31g!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dbd33dd-3f25-44ee-9bba-6e015bec2dc0_1536x1154.png 1272w, https://substackcdn.com/image/fetch/$s_!v31g!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dbd33dd-3f25-44ee-9bba-6e015bec2dc0_1536x1154.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!v31g!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dbd33dd-3f25-44ee-9bba-6e015bec2dc0_1536x1154.png" width="1456" height="1094" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4dbd33dd-3f25-44ee-9bba-6e015bec2dc0_1536x1154.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1094,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Slide: \&quot;ML estimates are severely biased in small samples.\&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Slide: &quot;ML estimates are severely biased in small samples.&quot;" title="Slide: &quot;ML estimates are severely biased in small samples.&quot;" srcset="https://substackcdn.com/image/fetch/$s_!v31g!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dbd33dd-3f25-44ee-9bba-6e015bec2dc0_1536x1154.png 424w, https://substackcdn.com/image/fetch/$s_!v31g!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dbd33dd-3f25-44ee-9bba-6e015bec2dc0_1536x1154.png 848w, https://substackcdn.com/image/fetch/$s_!v31g!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dbd33dd-3f25-44ee-9bba-6e015bec2dc0_1536x1154.png 1272w, https://substackcdn.com/image/fetch/$s_!v31g!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4dbd33dd-3f25-44ee-9bba-6e015bec2dc0_1536x1154.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>The figure below shows the percent bias in the coefficient estimates for different constants and numbers of explanatory variables (k) as the sample size varies. It&#8217;s hardly negligible, but it disappears quickly.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!WcOc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fa28cf8-cb3c-4685-b043-20e472df2bb1_1524x1138.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!WcOc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fa28cf8-cb3c-4685-b043-20e472df2bb1_1524x1138.png 424w, https://substackcdn.com/image/fetch/$s_!WcOc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fa28cf8-cb3c-4685-b043-20e472df2bb1_1524x1138.png 848w, https://substackcdn.com/image/fetch/$s_!WcOc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fa28cf8-cb3c-4685-b043-20e472df2bb1_1524x1138.png 1272w, https://substackcdn.com/image/fetch/$s_!WcOc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fa28cf8-cb3c-4685-b043-20e472df2bb1_1524x1138.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!WcOc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fa28cf8-cb3c-4685-b043-20e472df2bb1_1524x1138.png" width="1456" height="1087" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1fa28cf8-cb3c-4685-b043-20e472df2bb1_1524x1138.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1087,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Line chart titled \&quot;Percent Bias of the ML Estimator,\&quot; with nine panels for three intercepts and three numbers of explanatory variables. Bias falls quickly as the sample size grows from 50 to 200.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Line chart titled &quot;Percent Bias of the ML Estimator,&quot; with nine panels for three intercepts and three numbers of explanatory variables. Bias falls quickly as the sample size grows from 50 to 200." title="Line chart titled &quot;Percent Bias of the ML Estimator,&quot; with nine panels for three intercepts and three numbers of explanatory variables. Bias falls quickly as the sample size grows from 50 to 200." srcset="https://substackcdn.com/image/fetch/$s_!WcOc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fa28cf8-cb3c-4685-b043-20e472df2bb1_1524x1138.png 424w, https://substackcdn.com/image/fetch/$s_!WcOc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fa28cf8-cb3c-4685-b043-20e472df2bb1_1524x1138.png 848w, https://substackcdn.com/image/fetch/$s_!WcOc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fa28cf8-cb3c-4685-b043-20e472df2bb1_1524x1138.png 1272w, https://substackcdn.com/image/fetch/$s_!WcOc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fa28cf8-cb3c-4685-b043-20e472df2bb1_1524x1138.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>Fortunately, David Firth came along and <a href="https://www.jstor.org/stable/2336755">suggested a </a><em><a href="https://www.jstor.org/stable/2336755">penalized</a></em><a href="https://www.jstor.org/stable/2336755"> maximum likelihood estimator</a> that eliminates almost all of this bias.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5ww9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9fdf37-cdd5-4108-b847-76d75581f0a0_1492x956.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5ww9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9fdf37-cdd5-4108-b847-76d75581f0a0_1492x956.png 424w, https://substackcdn.com/image/fetch/$s_!5ww9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9fdf37-cdd5-4108-b847-76d75581f0a0_1492x956.png 848w, https://substackcdn.com/image/fetch/$s_!5ww9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9fdf37-cdd5-4108-b847-76d75581f0a0_1492x956.png 1272w, https://substackcdn.com/image/fetch/$s_!5ww9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9fdf37-cdd5-4108-b847-76d75581f0a0_1492x956.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5ww9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9fdf37-cdd5-4108-b847-76d75581f0a0_1492x956.png" width="1456" height="933" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7c9fdf37-cdd5-4108-b847-76d75581f0a0_1492x956.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:933,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;First page of David Firth's 1993 Biometrika article, \&quot;Bias reduction of maximum likelihood estimates.\&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="First page of David Firth's 1993 Biometrika article, &quot;Bias reduction of maximum likelihood estimates.&quot;" title="First page of David Firth's 1993 Biometrika article, &quot;Bias reduction of maximum likelihood estimates.&quot;" srcset="https://substackcdn.com/image/fetch/$s_!5ww9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9fdf37-cdd5-4108-b847-76d75581f0a0_1492x956.png 424w, https://substackcdn.com/image/fetch/$s_!5ww9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9fdf37-cdd5-4108-b847-76d75581f0a0_1492x956.png 848w, https://substackcdn.com/image/fetch/$s_!5ww9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9fdf37-cdd5-4108-b847-76d75581f0a0_1492x956.png 1272w, https://substackcdn.com/image/fetch/$s_!5ww9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9fdf37-cdd5-4108-b847-76d75581f0a0_1492x956.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>If this seems familiar, it should. Zorn&#8217;s (<a href="https://x.com/prisonrodeo">@prisonrodeo</a>) <a href="https://www.cambridge.org/core/journals/political-analysis/article/abs/solution-to-separation-in-binary-response-models/669DAA8BC4443AE56DF0F92113B47795">(2005) paper</a> is a classic in political science methods classes, and he recommends Firth&#8217;s penalty to deal with separation.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zt56!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2b1439e-eb6d-4d01-9596-f9464b19efbe_1572x1168.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zt56!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2b1439e-eb6d-4d01-9596-f9464b19efbe_1572x1168.png 424w, https://substackcdn.com/image/fetch/$s_!zt56!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2b1439e-eb6d-4d01-9596-f9464b19efbe_1572x1168.png 848w, https://substackcdn.com/image/fetch/$s_!zt56!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2b1439e-eb6d-4d01-9596-f9464b19efbe_1572x1168.png 1272w, https://substackcdn.com/image/fetch/$s_!zt56!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2b1439e-eb6d-4d01-9596-f9464b19efbe_1572x1168.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zt56!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2b1439e-eb6d-4d01-9596-f9464b19efbe_1572x1168.png" width="1456" height="1082" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a2b1439e-eb6d-4d01-9596-f9464b19efbe_1572x1168.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1082,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;First page of Christopher Zorn's article \&quot;A Solution to Separation in Binary Response Models.\&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="First page of Christopher Zorn's article &quot;A Solution to Separation in Binary Response Models.&quot;" title="First page of Christopher Zorn's article &quot;A Solution to Separation in Binary Response Models.&quot;" srcset="https://substackcdn.com/image/fetch/$s_!zt56!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2b1439e-eb6d-4d01-9596-f9464b19efbe_1572x1168.png 424w, https://substackcdn.com/image/fetch/$s_!zt56!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2b1439e-eb6d-4d01-9596-f9464b19efbe_1572x1168.png 848w, https://substackcdn.com/image/fetch/$s_!zt56!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2b1439e-eb6d-4d01-9596-f9464b19efbe_1572x1168.png 1272w, https://substackcdn.com/image/fetch/$s_!zt56!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2b1439e-eb6d-4d01-9596-f9464b19efbe_1572x1168.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>Here&#8217;s what Firth&#8217;s penalty looks like. You just maximize the penalized likelihood L* rather than the usual likelihood L.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!LU49!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45f7bec4-b27a-4b2c-a859-d692ceb3983b_1468x600.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!LU49!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45f7bec4-b27a-4b2c-a859-d692ceb3983b_1468x600.png 424w, https://substackcdn.com/image/fetch/$s_!LU49!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45f7bec4-b27a-4b2c-a859-d692ceb3983b_1468x600.png 848w, https://substackcdn.com/image/fetch/$s_!LU49!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45f7bec4-b27a-4b2c-a859-d692ceb3983b_1468x600.png 1272w, https://substackcdn.com/image/fetch/$s_!LU49!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45f7bec4-b27a-4b2c-a859-d692ceb3983b_1468x600.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!LU49!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45f7bec4-b27a-4b2c-a859-d692ceb3983b_1468x600.png" width="1456" height="595" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/45f7bec4-b27a-4b2c-a859-d692ceb3983b_1468x600.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:595,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Equation: the penalized likelihood L-star of beta given y equals the likelihood L of beta given y times the square root of the determinant of the information matrix, which is labeled \&quot;penalty.\&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Equation: the penalized likelihood L-star of beta given y equals the likelihood L of beta given y times the square root of the determinant of the information matrix, which is labeled &quot;penalty.&quot;" title="Equation: the penalized likelihood L-star of beta given y equals the likelihood L of beta given y times the square root of the determinant of the information matrix, which is labeled &quot;penalty.&quot;" srcset="https://substackcdn.com/image/fetch/$s_!LU49!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45f7bec4-b27a-4b2c-a859-d692ceb3983b_1468x600.png 424w, https://substackcdn.com/image/fetch/$s_!LU49!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45f7bec4-b27a-4b2c-a859-d692ceb3983b_1468x600.png 848w, https://substackcdn.com/image/fetch/$s_!LU49!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45f7bec4-b27a-4b2c-a859-d692ceb3983b_1468x600.png 1272w, https://substackcdn.com/image/fetch/$s_!LU49!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45f7bec4-b27a-4b2c-a859-d692ceb3983b_1468x600.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>And it really works! Here&#8217;s a comparison of the percent bias in the ML and PML estimators. You&#8217;ll see that Firth&#8217;s penalty just wipes most of the bias away.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!WwO5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc66b5fd7-e478-4284-92e3-f2c58a7180b2_1534x1154.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!WwO5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc66b5fd7-e478-4284-92e3-f2c58a7180b2_1534x1154.png 424w, https://substackcdn.com/image/fetch/$s_!WwO5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc66b5fd7-e478-4284-92e3-f2c58a7180b2_1534x1154.png 848w, https://substackcdn.com/image/fetch/$s_!WwO5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc66b5fd7-e478-4284-92e3-f2c58a7180b2_1534x1154.png 1272w, https://substackcdn.com/image/fetch/$s_!WwO5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc66b5fd7-e478-4284-92e3-f2c58a7180b2_1534x1154.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!WwO5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc66b5fd7-e478-4284-92e3-f2c58a7180b2_1534x1154.png" width="1456" height="1095" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c66b5fd7-e478-4284-92e3-f2c58a7180b2_1534x1154.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1095,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Line chart titled \&quot;Percent Bias of ML and PML Estimators,\&quot; with nine panels. ML bias starts as high as 120% with 30 observations (about 8 events) and falls with sample size; PML bias stays near zero (7% in that same case).&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Line chart titled &quot;Percent Bias of ML and PML Estimators,&quot; with nine panels. ML bias starts as high as 120% with 30 observations (about 8 events) and falls with sample size; PML bias stays near zero (7% in that same case)." title="Line chart titled &quot;Percent Bias of ML and PML Estimators,&quot; with nine panels. ML bias starts as high as 120% with 30 observations (about 8 events) and falls with sample size; PML bias stays near zero (7% in that same case)." srcset="https://substackcdn.com/image/fetch/$s_!WwO5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc66b5fd7-e478-4284-92e3-f2c58a7180b2_1534x1154.png 424w, https://substackcdn.com/image/fetch/$s_!WwO5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc66b5fd7-e478-4284-92e3-f2c58a7180b2_1534x1154.png 848w, https://substackcdn.com/image/fetch/$s_!WwO5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc66b5fd7-e478-4284-92e3-f2c58a7180b2_1534x1154.png 1272w, https://substackcdn.com/image/fetch/$s_!WwO5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc66b5fd7-e478-4284-92e3-f2c58a7180b2_1534x1154.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>BUT WAIT!!!! &#128721;</p><p>If you&#8217;re clever, you&#8217;ll ask about variance. Most of the time, when you reduce bias, you increase variance. You have to choose!</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!o3NF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc5e7eb-7dac-4d3a-9054-944ec9b64668_500x756.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!o3NF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc5e7eb-7dac-4d3a-9054-944ec9b64668_500x756.jpeg 424w, https://substackcdn.com/image/fetch/$s_!o3NF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc5e7eb-7dac-4d3a-9054-944ec9b64668_500x756.jpeg 848w, https://substackcdn.com/image/fetch/$s_!o3NF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc5e7eb-7dac-4d3a-9054-944ec9b64668_500x756.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!o3NF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc5e7eb-7dac-4d3a-9054-944ec9b64668_500x756.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!o3NF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc5e7eb-7dac-4d3a-9054-944ec9b64668_500x756.jpeg" width="500" height="756" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bbc5e7eb-7dac-4d3a-9054-944ec9b64668_500x756.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:756,&quot;width&quot;:500,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Meme: the two-buttons cartoon. A sweating man can't choose between buttons labeled \&quot;bias\&quot; and \&quot;variance.\&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Meme: the two-buttons cartoon. A sweating man can't choose between buttons labeled &quot;bias&quot; and &quot;variance.&quot;" title="Meme: the two-buttons cartoon. A sweating man can't choose between buttons labeled &quot;bias&quot; and &quot;variance.&quot;" srcset="https://substackcdn.com/image/fetch/$s_!o3NF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc5e7eb-7dac-4d3a-9054-944ec9b64668_500x756.jpeg 424w, https://substackcdn.com/image/fetch/$s_!o3NF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc5e7eb-7dac-4d3a-9054-944ec9b64668_500x756.jpeg 848w, https://substackcdn.com/image/fetch/$s_!o3NF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc5e7eb-7dac-4d3a-9054-944ec9b64668_500x756.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!o3NF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc5e7eb-7dac-4d3a-9054-944ec9b64668_500x756.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>But that&#8217;s not what happens here.</p><p>When you use Firth&#8217;s logit, you shrink <em>both</em> bias and variance.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!u6Mz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff9dd4f-b2db-4ee3-b563-19755cd0ee68_534x467.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!u6Mz!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff9dd4f-b2db-4ee3-b563-19755cd0ee68_534x467.jpeg 424w, https://substackcdn.com/image/fetch/$s_!u6Mz!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff9dd4f-b2db-4ee3-b563-19755cd0ee68_534x467.jpeg 848w, https://substackcdn.com/image/fetch/$s_!u6Mz!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff9dd4f-b2db-4ee3-b563-19755cd0ee68_534x467.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!u6Mz!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff9dd4f-b2db-4ee3-b563-19755cd0ee68_534x467.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!u6Mz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff9dd4f-b2db-4ee3-b563-19755cd0ee68_534x467.jpeg" width="534" height="467" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2ff9dd4f-b2db-4ee3-b563-19755cd0ee68_534x467.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:467,&quot;width&quot;:534,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Meme: the \&quot;Ancient Aliens\&quot; guy. \&quot;Firth's penalty reduces bias AND variance?! Aliens.\&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Meme: the &quot;Ancient Aliens&quot; guy. &quot;Firth's penalty reduces bias AND variance?! Aliens.&quot;" title="Meme: the &quot;Ancient Aliens&quot; guy. &quot;Firth's penalty reduces bias AND variance?! Aliens.&quot;" srcset="https://substackcdn.com/image/fetch/$s_!u6Mz!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff9dd4f-b2db-4ee3-b563-19755cd0ee68_534x467.jpeg 424w, https://substackcdn.com/image/fetch/$s_!u6Mz!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff9dd4f-b2db-4ee3-b563-19755cd0ee68_534x467.jpeg 848w, https://substackcdn.com/image/fetch/$s_!u6Mz!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff9dd4f-b2db-4ee3-b563-19755cd0ee68_534x467.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!u6Mz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff9dd4f-b2db-4ee3-b563-19755cd0ee68_534x467.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>That means you don&#8217;t have to choose between bias and variance. You can reduce BOTH.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Z6xR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c5a1fa1-fbde-4eb4-a44f-71c318f5c483_500x756.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Z6xR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c5a1fa1-fbde-4eb4-a44f-71c318f5c483_500x756.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Z6xR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c5a1fa1-fbde-4eb4-a44f-71c318f5c483_500x756.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Z6xR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c5a1fa1-fbde-4eb4-a44f-71c318f5c483_500x756.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Z6xR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c5a1fa1-fbde-4eb4-a44f-71c318f5c483_500x756.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Z6xR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c5a1fa1-fbde-4eb4-a44f-71c318f5c483_500x756.jpeg" width="500" height="756" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9c5a1fa1-fbde-4eb4-a44f-71c318f5c483_500x756.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:756,&quot;width&quot;:500,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Meme: the two-buttons cartoon, edited so one hand presses both \&quot;bias\&quot; and \&quot;variance\&quot; while the man gives a happy thumbs up.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Meme: the two-buttons cartoon, edited so one hand presses both &quot;bias&quot; and &quot;variance&quot; while the man gives a happy thumbs up." title="Meme: the two-buttons cartoon, edited so one hand presses both &quot;bias&quot; and &quot;variance&quot; while the man gives a happy thumbs up." srcset="https://substackcdn.com/image/fetch/$s_!Z6xR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c5a1fa1-fbde-4eb4-a44f-71c318f5c483_500x756.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Z6xR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c5a1fa1-fbde-4eb4-a44f-71c318f5c483_500x756.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Z6xR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c5a1fa1-fbde-4eb4-a44f-71c318f5c483_500x756.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Z6xR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c5a1fa1-fbde-4eb4-a44f-71c318f5c483_500x756.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>Here&#8217;s a figure showing how much more variable your estimates will be if you use ML rather than Firth&#8217;s PML.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!bb0M!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa013c1f9-b2db-4152-a378-00e87528a8b6_1538x1154.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!bb0M!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa013c1f9-b2db-4152-a378-00e87528a8b6_1538x1154.png 424w, https://substackcdn.com/image/fetch/$s_!bb0M!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa013c1f9-b2db-4152-a378-00e87528a8b6_1538x1154.png 848w, https://substackcdn.com/image/fetch/$s_!bb0M!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa013c1f9-b2db-4152-a378-00e87528a8b6_1538x1154.png 1272w, https://substackcdn.com/image/fetch/$s_!bb0M!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa013c1f9-b2db-4152-a378-00e87528a8b6_1538x1154.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!bb0M!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa013c1f9-b2db-4152-a378-00e87528a8b6_1538x1154.png" width="1456" height="1092" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a013c1f9-b2db-4152-a378-00e87528a8b6_1538x1154.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1092,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Line chart titled \&quot;Variance Inflation (%) of ML Relative to PML,\&quot; with nine panels. ML's extra variance is largest in small samples and shrinks as the sample size grows.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Line chart titled &quot;Variance Inflation (%) of ML Relative to PML,&quot; with nine panels. ML's extra variance is largest in small samples and shrinks as the sample size grows." title="Line chart titled &quot;Variance Inflation (%) of ML Relative to PML,&quot; with nine panels. ML's extra variance is largest in small samples and shrinks as the sample size grows." srcset="https://substackcdn.com/image/fetch/$s_!bb0M!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa013c1f9-b2db-4152-a378-00e87528a8b6_1538x1154.png 424w, https://substackcdn.com/image/fetch/$s_!bb0M!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa013c1f9-b2db-4152-a378-00e87528a8b6_1538x1154.png 848w, https://substackcdn.com/image/fetch/$s_!bb0M!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa013c1f9-b2db-4152-a378-00e87528a8b6_1538x1154.png 1272w, https://substackcdn.com/image/fetch/$s_!bb0M!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa013c1f9-b2db-4152-a378-00e87528a8b6_1538x1154.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>But even more importantly, it turns out that bias isn&#8217;t the big problem in the first place. The shrinkage in the variance is much more important than the reduction in bias.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!qjv9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F210de859-ad69-4303-91f4-6e954e110005_576x433.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!qjv9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F210de859-ad69-4303-91f4-6e954e110005_576x433.jpeg 424w, https://substackcdn.com/image/fetch/$s_!qjv9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F210de859-ad69-4303-91f4-6e954e110005_576x433.jpeg 848w, https://substackcdn.com/image/fetch/$s_!qjv9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F210de859-ad69-4303-91f4-6e954e110005_576x433.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!qjv9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F210de859-ad69-4303-91f4-6e954e110005_576x433.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!qjv9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F210de859-ad69-4303-91f4-6e954e110005_576x433.jpeg" width="576" height="433" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/210de859-ad69-4303-91f4-6e954e110005_576x433.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:433,&quot;width&quot;:576,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Meme: Jimmy McMillan, \&quot;The variance is too damn high!\&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Meme: Jimmy McMillan, &quot;The variance is too damn high!&quot;" title="Meme: Jimmy McMillan, &quot;The variance is too damn high!&quot;" srcset="https://substackcdn.com/image/fetch/$s_!qjv9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F210de859-ad69-4303-91f4-6e954e110005_576x433.jpeg 424w, https://substackcdn.com/image/fetch/$s_!qjv9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F210de859-ad69-4303-91f4-6e954e110005_576x433.jpeg 848w, https://substackcdn.com/image/fetch/$s_!qjv9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F210de859-ad69-4303-91f4-6e954e110005_576x433.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!qjv9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F210de859-ad69-4303-91f4-6e954e110005_576x433.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>In many common scenarios, the variance might contribute about 25 times more to the MSE than the bias (or higher).</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!axQb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F760dcd06-7a82-42fd-b05e-05f305e2a2b8_1542x1160.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!axQb!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F760dcd06-7a82-42fd-b05e-05f305e2a2b8_1542x1160.png 424w, https://substackcdn.com/image/fetch/$s_!axQb!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F760dcd06-7a82-42fd-b05e-05f305e2a2b8_1542x1160.png 848w, https://substackcdn.com/image/fetch/$s_!axQb!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F760dcd06-7a82-42fd-b05e-05f305e2a2b8_1542x1160.png 1272w, https://substackcdn.com/image/fetch/$s_!axQb!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F760dcd06-7a82-42fd-b05e-05f305e2a2b8_1542x1160.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!axQb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F760dcd06-7a82-42fd-b05e-05f305e2a2b8_1542x1160.png" width="1456" height="1095" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/760dcd06-7a82-42fd-b05e-05f305e2a2b8_1542x1160.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1095,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Scatterplot titled \&quot;The Relative Contribution of the Variance Compared to the Bias as the Sample Size Varies,\&quot; by number of variables and intercept. The variance's share relative to the bias grows with sample size, from under 10 times at N = 30 to as much as 180 times at N = 210.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Scatterplot titled &quot;The Relative Contribution of the Variance Compared to the Bias as the Sample Size Varies,&quot; by number of variables and intercept. The variance's share relative to the bias grows with sample size, from under 10 times at N = 30 to as much as 180 times at N = 210." title="Scatterplot titled &quot;The Relative Contribution of the Variance Compared to the Bias as the Sample Size Varies,&quot; by number of variables and intercept. The variance's share relative to the bias grows with sample size, from under 10 times at N = 30 to as much as 180 times at N = 210." srcset="https://substackcdn.com/image/fetch/$s_!axQb!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F760dcd06-7a82-42fd-b05e-05f305e2a2b8_1542x1160.png 424w, https://substackcdn.com/image/fetch/$s_!axQb!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F760dcd06-7a82-42fd-b05e-05f305e2a2b8_1542x1160.png 848w, https://substackcdn.com/image/fetch/$s_!axQb!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F760dcd06-7a82-42fd-b05e-05f305e2a2b8_1542x1160.png 1272w, https://substackcdn.com/image/fetch/$s_!axQb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F760dcd06-7a82-42fd-b05e-05f305e2a2b8_1542x1160.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>So you shouldn&#8217;t really be using PML to reduce bias; you should be using PML to reduce <em>variance</em> (and bias).</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!jfsf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c9eec2e-7a5a-4717-b892-f22e540f1231_687x500.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!jfsf!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c9eec2e-7a5a-4717-b892-f22e540f1231_687x500.jpeg 424w, https://substackcdn.com/image/fetch/$s_!jfsf!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c9eec2e-7a5a-4717-b892-f22e540f1231_687x500.jpeg 848w, https://substackcdn.com/image/fetch/$s_!jfsf!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c9eec2e-7a5a-4717-b892-f22e540f1231_687x500.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!jfsf!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c9eec2e-7a5a-4717-b892-f22e540f1231_687x500.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!jfsf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c9eec2e-7a5a-4717-b892-f22e540f1231_687x500.jpeg" width="687" height="500" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1c9eec2e-7a5a-4717-b892-f22e540f1231_687x500.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:500,&quot;width&quot;:687,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Meme: Tuxedo Winnie the Pooh. Plain Pooh: \&quot;Uses PML to reduce bias.\&quot; Pooh in a tuxedo: \&quot;Uses PML to reduce variance.\&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Meme: Tuxedo Winnie the Pooh. Plain Pooh: &quot;Uses PML to reduce bias.&quot; Pooh in a tuxedo: &quot;Uses PML to reduce variance.&quot;" title="Meme: Tuxedo Winnie the Pooh. Plain Pooh: &quot;Uses PML to reduce bias.&quot; Pooh in a tuxedo: &quot;Uses PML to reduce variance.&quot;" srcset="https://substackcdn.com/image/fetch/$s_!jfsf!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c9eec2e-7a5a-4717-b892-f22e540f1231_687x500.jpeg 424w, https://substackcdn.com/image/fetch/$s_!jfsf!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c9eec2e-7a5a-4717-b892-f22e540f1231_687x500.jpeg 848w, https://substackcdn.com/image/fetch/$s_!jfsf!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c9eec2e-7a5a-4717-b892-f22e540f1231_687x500.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!jfsf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c9eec2e-7a5a-4717-b892-f22e540f1231_687x500.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>All of this means that you should usually use <em>penalized</em> maximum likelihood to fit logistic regression models.</p><p>As a default, Firth&#8217;s penalty makes much more sense than the usual maximum likelihood estimator.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!hzqF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4133592-32c1-496b-a430-7c597e87f1c1_500x671.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!hzqF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4133592-32c1-496b-a430-7c597e87f1c1_500x671.jpeg 424w, https://substackcdn.com/image/fetch/$s_!hzqF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4133592-32c1-496b-a430-7c597e87f1c1_500x671.jpeg 848w, https://substackcdn.com/image/fetch/$s_!hzqF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4133592-32c1-496b-a430-7c597e87f1c1_500x671.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!hzqF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4133592-32c1-496b-a430-7c597e87f1c1_500x671.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!hzqF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4133592-32c1-496b-a430-7c597e87f1c1_500x671.jpeg" width="500" height="671" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e4133592-32c1-496b-a430-7c597e87f1c1_500x671.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:671,&quot;width&quot;:500,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Meme: Uncle Sam pointing. \&quot;I need you to use penalized maximum likelihood.\&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Meme: Uncle Sam pointing. &quot;I need you to use penalized maximum likelihood.&quot;" title="Meme: Uncle Sam pointing. &quot;I need you to use penalized maximum likelihood.&quot;" srcset="https://substackcdn.com/image/fetch/$s_!hzqF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4133592-32c1-496b-a430-7c597e87f1c1_500x671.jpeg 424w, https://substackcdn.com/image/fetch/$s_!hzqF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4133592-32c1-496b-a430-7c597e87f1c1_500x671.jpeg 848w, https://substackcdn.com/image/fetch/$s_!hzqF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4133592-32c1-496b-a430-7c597e87f1c1_500x671.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!hzqF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4133592-32c1-496b-a430-7c597e87f1c1_500x671.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>In practice, that means using the <a href="https://cran.r-project.org/package=brglm2">{brglm2}</a> package rather than glm().</p><p>And Twitter will love this! {brglm2} works with <a href="https://x.com/VincentAB">@VincentAB</a>&#8217;s <a href="https://marginaleffects.com/">{marginaleffects}</a> package and <a href="https://x.com/noah_greifer">@noah_greifer</a>&#8217;s <a href="https://iqss.github.io/clarify/">{clarify}</a> package.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!JV3s!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d672c56-ce31-4df6-bc65-301fd94da199_3144x1650.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!JV3s!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d672c56-ce31-4df6-bc65-301fd94da199_3144x1650.png 424w, https://substackcdn.com/image/fetch/$s_!JV3s!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d672c56-ce31-4df6-bc65-301fd94da199_3144x1650.png 848w, https://substackcdn.com/image/fetch/$s_!JV3s!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d672c56-ce31-4df6-bc65-301fd94da199_3144x1650.png 1272w, https://substackcdn.com/image/fetch/$s_!JV3s!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d672c56-ce31-4df6-bc65-301fd94da199_3144x1650.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!JV3s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d672c56-ce31-4df6-bc65-301fd94da199_3144x1650.png" width="1456" height="764" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3d672c56-ce31-4df6-bc65-301fd94da199_3144x1650.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:764,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Meme: Drake rejects glm(..., family = \&quot;binomial\&quot;) and approves glm(..., family = \&quot;binomial\&quot;, method = \&quot;brglmFit\&quot;).&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Meme: Drake rejects glm(..., family = &quot;binomial&quot;) and approves glm(..., family = &quot;binomial&quot;, method = &quot;brglmFit&quot;)." title="Meme: Drake rejects glm(..., family = &quot;binomial&quot;) and approves glm(..., family = &quot;binomial&quot;, method = &quot;brglmFit&quot;)." srcset="https://substackcdn.com/image/fetch/$s_!JV3s!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d672c56-ce31-4df6-bc65-301fd94da199_3144x1650.png 424w, https://substackcdn.com/image/fetch/$s_!JV3s!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d672c56-ce31-4df6-bc65-301fd94da199_3144x1650.png 848w, https://substackcdn.com/image/fetch/$s_!JV3s!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d672c56-ce31-4df6-bc65-301fd94da199_3144x1650.png 1272w, https://substackcdn.com/image/fetch/$s_!JV3s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d672c56-ce31-4df6-bc65-301fd94da199_3144x1650.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>And it can make a big difference! Here&#8217;s a comparison for a small data set from Weisiger (2014).</p><p>Paper <a href="https://journals.sagepub.com/doi/pdf/10.1177/0738894213508691">here</a>.</p><p>Code <a href="https://gist.github.com/carlislerainey/61c7c7693f6ffa852981c7c981ad62cb">here</a>.</p><div><hr></div><p>Here&#8217;s the plot</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xsT6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8af1b24d-a114-41d5-88f6-ba09fc9d593f_725x415.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xsT6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8af1b24d-a114-41d5-88f6-ba09fc9d593f_725x415.png 424w, https://substackcdn.com/image/fetch/$s_!xsT6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8af1b24d-a114-41d5-88f6-ba09fc9d593f_725x415.png 848w, https://substackcdn.com/image/fetch/$s_!xsT6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8af1b24d-a114-41d5-88f6-ba09fc9d593f_725x415.png 1272w, https://substackcdn.com/image/fetch/$s_!xsT6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8af1b24d-a114-41d5-88f6-ba09fc9d593f_725x415.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xsT6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8af1b24d-a114-41d5-88f6-ba09fc9d593f_725x415.png" width="725" height="415" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8af1b24d-a114-41d5-88f6-ba09fc9d593f_725x415.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:415,&quot;width&quot;:725,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Coefficient plot comparing ML and PML estimates with 95% confidence intervals for seven terms in the Weisiger (2014) model. The PML intervals are narrower and the estimates closer to zero.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Coefficient plot comparing ML and PML estimates with 95% confidence intervals for seven terms in the Weisiger (2014) model. The PML intervals are narrower and the estimates closer to zero." title="Coefficient plot comparing ML and PML estimates with 95% confidence intervals for seven terms in the Weisiger (2014) model. The PML intervals are narrower and the estimates closer to zero." srcset="https://substackcdn.com/image/fetch/$s_!xsT6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8af1b24d-a114-41d5-88f6-ba09fc9d593f_725x415.png 424w, https://substackcdn.com/image/fetch/$s_!xsT6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8af1b24d-a114-41d5-88f6-ba09fc9d593f_725x415.png 848w, https://substackcdn.com/image/fetch/$s_!xsT6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8af1b24d-a114-41d5-88f6-ba09fc9d593f_725x415.png 1272w, https://substackcdn.com/image/fetch/$s_!xsT6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8af1b24d-a114-41d5-88f6-ba09fc9d593f_725x415.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>In short, I think Firth&#8217;s PML is usually preferable to ML for fitting logit models. It&#8217;s always better in theory (smaller bias and variance), easy to implement (brglm2), makes BIG difference in small samples, and a meaningful difference in much larger samples (e.g., N = 1,000).</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!aO0J!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd55210b7-0c3b-43aa-9563-181c123bc398_1512x1038.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!aO0J!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd55210b7-0c3b-43aa-9563-181c123bc398_1512x1038.png 424w, https://substackcdn.com/image/fetch/$s_!aO0J!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd55210b7-0c3b-43aa-9563-181c123bc398_1512x1038.png 848w, https://substackcdn.com/image/fetch/$s_!aO0J!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd55210b7-0c3b-43aa-9563-181c123bc398_1512x1038.png 1272w, https://substackcdn.com/image/fetch/$s_!aO0J!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd55210b7-0c3b-43aa-9563-181c123bc398_1512x1038.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!aO0J!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd55210b7-0c3b-43aa-9563-181c123bc398_1512x1038.png" width="1456" height="1000" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d55210b7-0c3b-43aa-9563-181c123bc398_1512x1038.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1000,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Slide titled \&quot;PML in Short\&quot;: Always better in theory. Easy to implement. Makes a big difference in small samples. Makes a small, but noticeable, difference in much larger samples.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Slide titled &quot;PML in Short&quot;: Always better in theory. Easy to implement. Makes a big difference in small samples. Makes a small, but noticeable, difference in much larger samples." title="Slide titled &quot;PML in Short&quot;: Always better in theory. Easy to implement. Makes a big difference in small samples. Makes a small, but noticeable, difference in much larger samples." srcset="https://substackcdn.com/image/fetch/$s_!aO0J!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd55210b7-0c3b-43aa-9563-181c123bc398_1512x1038.png 424w, https://substackcdn.com/image/fetch/$s_!aO0J!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd55210b7-0c3b-43aa-9563-181c123bc398_1512x1038.png 848w, https://substackcdn.com/image/fetch/$s_!aO0J!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd55210b7-0c3b-43aa-9563-181c123bc398_1512x1038.png 1272w, https://substackcdn.com/image/fetch/$s_!aO0J!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd55210b7-0c3b-43aa-9563-181c123bc398_1512x1038.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>If you&#8217;re interested in this topic, then I recommend the work of <a href="https://www.ikosmidis.com/">Ioannis Kosmidis</a> (<a href="https://x.com/IKosmidis_">@IKosmidis_</a>).</p><div><hr></div><p>And here&#8217;s a nugget for #econtwitter. For a simple treatment/control design with a binary outcome, Firth&#8217;s logit produces a better estimate of the ATE than OLS.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!KI1n!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25978a69-55b5-4e8d-8eee-b0528d61c1fe_480x324.gif" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!KI1n!,w_424,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25978a69-55b5-4e8d-8eee-b0528d61c1fe_480x324.gif 424w, https://substackcdn.com/image/fetch/$s_!KI1n!,w_848,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25978a69-55b5-4e8d-8eee-b0528d61c1fe_480x324.gif 848w, https://substackcdn.com/image/fetch/$s_!KI1n!,w_1272,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25978a69-55b5-4e8d-8eee-b0528d61c1fe_480x324.gif 1272w, https://substackcdn.com/image/fetch/$s_!KI1n!,w_1456,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25978a69-55b5-4e8d-8eee-b0528d61c1fe_480x324.gif 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!KI1n!,w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25978a69-55b5-4e8d-8eee-b0528d61c1fe_480x324.gif" width="480" height="324" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/25978a69-55b5-4e8d-8eee-b0528d61c1fe_480x324.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:324,&quot;width&quot;:480,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Animated GIF: a baseball player in the dugout smiles, then his face turns to a startled stare.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Animated GIF: a baseball player in the dugout smiles, then his face turns to a startled stare." title="Animated GIF: a baseball player in the dugout smiles, then his face turns to a startled stare." srcset="https://substackcdn.com/image/fetch/$s_!KI1n!,w_424,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25978a69-55b5-4e8d-8eee-b0528d61c1fe_480x324.gif 424w, https://substackcdn.com/image/fetch/$s_!KI1n!,w_848,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25978a69-55b5-4e8d-8eee-b0528d61c1fe_480x324.gif 848w, https://substackcdn.com/image/fetch/$s_!KI1n!,w_1272,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25978a69-55b5-4e8d-8eee-b0528d61c1fe_480x324.gif 1272w, https://substackcdn.com/image/fetch/$s_!KI1n!,w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25978a69-55b5-4e8d-8eee-b0528d61c1fe_480x324.gif 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>Here are two takeaways.</p><p>1&#65039;&#8419; This &#8220;small sample&#8221; problem is a problem for even larger samples (perhaps larger than 1,000).</p><p>2&#65039;&#8419; The real problem isn&#8217;t bias; the problem is variance.</p><div><hr></div><p>If you&#8217;re interested, here&#8217;s <a href="https://www.cambridge.org/core/journals/political-science-research-and-methods/article/estimating-logit-models-with-small-samples/712F6039372475B650B123E9A1FE60E6">the paper</a> (with Kelly McCaskey) that describes all the details. It&#8217;s open access.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!2g7s!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21df878b-b6ce-4467-aca5-eb1ed1ac4d31_500x500.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2g7s!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21df878b-b6ce-4467-aca5-eb1ed1ac4d31_500x500.jpeg 424w, https://substackcdn.com/image/fetch/$s_!2g7s!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21df878b-b6ce-4467-aca5-eb1ed1ac4d31_500x500.jpeg 848w, https://substackcdn.com/image/fetch/$s_!2g7s!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21df878b-b6ce-4467-aca5-eb1ed1ac4d31_500x500.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!2g7s!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21df878b-b6ce-4467-aca5-eb1ed1ac4d31_500x500.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!2g7s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21df878b-b6ce-4467-aca5-eb1ed1ac4d31_500x500.jpeg" width="500" height="500" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/21df878b-b6ce-4467-aca5-eb1ed1ac4d31_500x500.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:500,&quot;width&quot;:500,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Meme: Anakin and Padm&#233;. Anakin: \&quot;I'm gonna fit a logit.\&quot; Padm&#233;: \&quot;You'll use PML, right?\&quot; Then, unsmiling: \&quot;Right?\&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Meme: Anakin and Padm&#233;. Anakin: &quot;I'm gonna fit a logit.&quot; Padm&#233;: &quot;You'll use PML, right?&quot; Then, unsmiling: &quot;Right?&quot;" title="Meme: Anakin and Padm&#233;. Anakin: &quot;I'm gonna fit a logit.&quot; Padm&#233;: &quot;You'll use PML, right?&quot; Then, unsmiling: &quot;Right?&quot;" srcset="https://substackcdn.com/image/fetch/$s_!2g7s!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21df878b-b6ce-4467-aca5-eb1ed1ac4d31_500x500.jpeg 424w, https://substackcdn.com/image/fetch/$s_!2g7s!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21df878b-b6ce-4467-aca5-eb1ed1ac4d31_500x500.jpeg 848w, https://substackcdn.com/image/fetch/$s_!2g7s!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21df878b-b6ce-4467-aca5-eb1ed1ac4d31_500x500.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!2g7s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21df878b-b6ce-4467-aca5-eb1ed1ac4d31_500x500.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div>]]></content:encoded></item></channel></rss>