C. Eric Hollis

C. Eric Hollis

Microsoft Threat Intelligence Python Security Tools.

msticpy is a library for InfoSec investigation and hunting in Jupyter Notebooks. It includes functionality to:

query log data from multiple sources
enrich the data with Threat Intelligence, geolocations and Azure resource data
extract Indicators of Activity (IoA) from logs and unpack encoded data
perform sophisticated analysis such as anomalous session detection and time series decomposition
visualize data using interactive timelines, process trees and multi-dimensional Morph Charts

It also includes some time-saving notebook tools such as widgets to set query time boundaries, select and display items from lists, and configure the notebook environment.

The msticpy package was initially developed to support Jupyter Notebooks authoring for Azure Sentinel. While Azure Sentinel is still a big focus of our work, we are extending the data query/acquisition components to pull log data from other sources (currently Splunk, Microsoft Defender for Endpoint and Microsoft Graph are supported but we are actively working on support for data from other SIEM platforms). Most of the components can also be used with data from any source. Pandas DataFrames are used as the ubiquitous input and output format of almost all components. There is also a data provider to make it easy to and process data from local CSV files and pickled DataFrames.

The package addresses three central needs for security investigators and hunters:

Acquiring and enriching data
Analyzing data
Visualizing data

We welcome feedback, bug reports, suggestions for new features and contributions.

Installing

For core install:

pip install msticpy

If you are using MSTICPy with Azure Sentinel you should install with the "azsentinel" extra package:

pip install msticpy[azsentinel]

or for the latest dev build

pip install git+https://github.com/microsoft/msticpy

Documentation

Full documentation is at ReadTheDocs

Sample notebooks for many of the modules are in the docs/notebooks folder and accompanying notebooks.

You can also browse through the sample notebooks referenced at the end of this document to see some of the functionality used in context. You can play with some of the package functions in this interactive demo on mybinder.org.

Log Data Acquisition

QueryProvider is an extensible query library targeting Azure Sentinel/Log Analytics, Splunk, OData and other log data sources. It also has special support for Mordor data sets and using local data.

Built-in parameterized queries allow complex queries to be run from a single function call. Add your own queries using a simple YAML schema.

Data Queries Notebook

Data Enrichment

Threat Intelligence providers

The TILookup class can lookup IoCs across multiple TI providers. built-in providers include AlienVault OTX, IBM XForce, VirusTotal and Azure Sentinel.

The input can be a single IoC observable or a pandas DataFrame containing multiple observables. Depending on the provider, you may require an account and an API key. Some providers also enforce throttling (especially for free tiers), which might affect performing bulk lookups.

TIProviders and TILookup Usage Notebook

GeoLocation Data

The GeoIP lookup classes allow you to match the geo-locations of IP addresses using either:

GeoLiteLookup - Maxmind Geolite (see https://www.maxmind.com)
IPStackLookup - IPStack (see https://ipstack.com)

GeoIP Lookup and GeoIP Notebook

Azure Resource Data, Storage and Azure Sentinel API

The AzureData module contains functionality for enriching data regarding Azure host details with additional host details exposed via the Azure API. The AzureSentinel module allows you to query incidents, retrieve detector and hunting queries. AzureBlogStorage lets you read and write data from blob storage.

Azure Resource APIs, Azure Sentinel APIs, Azure Storage

Security Analysis

This subpackage contains several modules helpful for working on security investigations and hunting:

Anomalous Sequence Detection

Detect unusual sequences of events in your Office, Active Directory or other log data. You can extract sessions (e.g. activity initiated by the same account) and identify and visualize unusual sequences of activity. For example, detecting an attacker setting a mail forwarding rule on someone's mailbox.

Anomalous Sessions and Anomalous Sequence Notebook

Time Series Analysis

Time series analysis allows you to identify unusual patterns in your log data taking into account normal seasonal variations (e.g. the regular ebb and flow of events over hours of the day, days of the week, etc.). Using both analysis and visualization highlights unusual traffic flows or event activity for any data set.

Time Series

Visualization

Event Timelines

Display any log events on an interactive timeline. Using the Bokeh Visualization Library the timeline control enables you to visualize one or more event streams, interactively zoom into specific time slots and view event details for plotted events.

Timeline and Timeline Notebook

Process Trees

The process tree functionality has two main components:

Process Tree creation - taking a process creation log from a host and building the parent-child relationships between processes in the data set.
Process Tree visualization - this takes the processed output displays an interactive process tree using Bokeh plots.

There are a set of utility functions to extract individual and partial trees from the processed data set.

Process Tree and Process Tree Notebook

Data Manipulation and Utility functions

Pivot Functions

Lets you use MSTICPy functionality in an "entity-centric" way. All functions, queries and lookups that relate to a particular entity type (e.g. Host, IpAddress, Url) are collected together as methods of that entity class. So, if you want to do things with an IP address, just load the IpAddress entity and browse its methods.

Pivot Functions and Pivot Functions Notebook

base64unpack

Base64 and archive (gz, zip, tar) extractor. It will try to identify any base64 encoded strings and try decode them. If the result looks like one of the supported archive types it will unpack the contents. The results of each decode/unpack are rechecked for further base64 content and up to a specified depth.

Base64 Decoding and Base64Unpack Notebook

iocextract

Uses regular expressions to look for Indicator of Compromise (IoC) patterns - IP Addresses, URLs, DNS domains, Hashes, file paths. Input can be a single string or a pandas dataframe.

IoC Extraction and IoCExtract Notebook

eventcluster (experimental)

This module is intended to be used to summarize large numbers of events into clusters of different patterns. High volume repeating events can often make it difficult to see unique and interesting items.

This is an unsupervised learning module implemented using SciKit Learn DBScan.

Event Clustering and Event Clustering Notebook

auditdextract

Module to load and decode Linux audit logs. It collapses messages sharing the same message ID into single events, decodes hex-encoded data fields and performs some event-specific formatting and normalization (e.g. for process start events it will re-assemble the process command line arguments into a single string).

syslog_utils

Module to support an investigation of a Linux host with only syslog logging enabled. This includes functions for collating host data, clustering logon events and detecting user sessions containing suspicious activity.

cmd_line

A module to support he detection of known malicious command line activity or suspicious patterns of command line activity.

domain_utils

A module to support investigation of domain names and URLs with functions to validate a domain name and screenshot a URL.

Notebook widgets

These are built from the Jupyter ipywidgets collection and group common functionality useful in InfoSec tasks such as list pickers, query time boundary settings and event display into an easy-to-use format.

More Notebooks on Azure Sentinel Notebooks GitHub

Azure Sentinel Notebooks

Example notebooks:

View directly on GitHub or copy and paste the link into nbviewer.org

Notebook examples with saved data

See the following notebooks for more examples of the use of this package in practice:

Windows Alert Investigation in GitHub or NbViewer
Office 365 Exploration in GitHub or NbViewer
Cross-Network Hunting in GitHubor NbViewer

Supported Platforms and Packages

msticpy is OS-independent
Requires Python 3.6 or later
See requirements.txt for more details and version requirements.

Contributing

For (brief) developer guidelines, see this wiki article Contributor Guidelines

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

Download Msticpy

"PHoss is a sniffer. A normal sniffer software is designed to find problems in data communication on the network. PHoss is designed to know some protocols which use (or may use) clear text passwords. Many protocols are designed to use secure authentication. For fallback they define a lowest level of authentication using clear text. Many companies use this lowest fallback definition as standard setting to make the product working in many environments." read more...

Download: http://www.phenoelit-us.org/phoss/download.html

More articles

C. Eric Hollis

Welcome back to Linux Command Line Hackery series, I hope you've enjoyed this series so far and would have learned something (at least a bit). Today we're going to get into user management, that is we are going to learn commands that will help us add and remove users and groups. So bring it on...

Before we get into adding new users to our system lets first talk about a command that will be useful if you are a non-root user.

Command: sudo
Syntax: sudo [options] command
Description: sudo allows a permitted user to execute a command as a superuser or another user.

Since the commands to follow need root privileges, if you are not root then don't forget to prefix these commands with sudo command. And yes you'll need to enter the root password in order to execute any command with sudo as root.

Command: useradd
Syntax: useradd [options] username
Description: this command is used for creating new user but is kinda old school.
Lets try to add a new user to our box.
[Note: I'm performing these commands as root user, you'll need root privileges to add a new user to your box. If you aren't root then you can try these commands by prefixing the sudo command at the very beginning of these command like this sudo useradd joe. You'll be prompted for your root password, enter it and you're good to go]

useradd joe

To verify that this command has really added a user to our box we can look at three files that store a users data on a Linux box, which are:

/etc/passwd -> this file stores information about a user separated by colons in this manner, first is login name, then in past there used to be an encrypted password hash at the second place however since the password hashes were moved to shadow file now it has a cross (x) there, then there is user id, after it is the user's group id, following it is a comment field, then the next field contains users home directory, and at last is the login shell of the user.

/etc/group -> this file stores information about groups, that is id of the group and to which group an user belongs.

/etc/shadow -> this file stores the encrypted password of users.

Using our command line techniques we learned so far lets check out these files and verify if our user has been created:

cat /etc/passwd /etc/group /etc/shadow | grep joe

In the above screenshot you can notice an ! in the /etc/shadow, this means the password of this user has not been set yet. That means we have to set the password of user joe manually, lets do just that.

Command: passwd
Syntax: passwd [options] [username]
Description: this command is used to change the password of user accounts.
Note that this command needs root privileges. So if you are not root then prefix this command with sudo.

passwd joe

After typing this command, you'll be prompted password and then for verifying your password. The password won't show up on the terminal.
Now joe's account is up and running with a password.

The useradd command is a old school command, lets create a new user with a different command which is kinda interactive.

Command: adduser
Syntax: adduser [options] user
Description: adduser command adds a user to the system. It is more friendly front-end to the useradd command.

So lets create a new user with adduser.

adduser jane

as seen in the image it prompts for password, full name and many other things and thus is easy to use.

OK now we know how to create a user its time to create a group which is very easy.

Command: addgroup
Syntax: addgroup [options] groupname
Description: This command is used to create a new group or add an existing user to an existing group.

We create a new group like this

addgroup grownups

So now we have a group called grownups, you can verify it by looking at /etc/group file.
Since joe is not a grownup user yet but jane is we'll add jane to grownups group like this:

addgroup jane grownups

Now jane is the member of grownups.

Its time to learn how to remove a user from our system and how to remove a group from the system, lets get straight to that.

Command: deluser
Syntax: deluser [options] username
Description: remove a user from system.

Lets remove joe from our system

deluser joe

Yes its as easy as that. But remember by default deluser will remove the user without removing the home directory or any other files owned by the user. Removing the home directory can be achieved by using the --remove-home option.

deluser jane --remove-home

Also the --remove-all-files option removes all the files from the system owned by the user (better watch-out). And to create a backup of all the files before deleting use the --backup option.

We don't need grownups group so lets remove it.

Command: delgroup
Syntax: delgroup [options] groupname
Description: remove a group from the system.

To remove grownups group just type:

delgroup grownups

That's it for today hope you got something in your head.

C. Eric Hollis

Friday, June 2, 2023

Msticpy - Microsoft Threat Intelligence Security Tools

Read more

PHoss: A Password Sniffer

Related articles

Thursday, June 1, 2023

Ettercap: Man In The Middle (MITM)

More articles

Linux Command Line Hackery Series - Part 6

More info

Followers

Blog Archive

About Me