The current demand for cognitive assessment cannot be met with traditional in-person methods, warranting the need for remote unsupervised options. However, lack of visibility into testing conditions and effort levels limit the utility of existing remote options. This retrospective study analyzed the frequency of and factors associated with environmental distractions during a brief digital assessment taken at home by 1,442 adults aged 23–84. Automated scoring algorithms flagged low data capture. Frequency of environmental distractions were manually counted on a per-frame and per-trial basis. A total of 7.4% of test administrations included distractions. Distractions were more frequent in men (41:350) than women (65:1,092) and the average age of distracted participants (51.7) was lower than undistracted participants (57.8). These results underscore the challenges associated with unsupervised cognitive assessment. Data collection methods that enable review of testing conditions are needed to confirm quality, usability, and actionability.