{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Обработка табличных данных" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "### Введение в Pandas \n", "\n", "Pandas - библиотека для работы с табличными данными в питоне." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Чтобы не писать название библиотеки целиком каждый раз, когда понадобится ее использовать, принято сокращать название библиотеки и импортировать ее как \"pd\":" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd # импортировали библиотеку pandas и назвали ее pd " ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "data = pd.read_csv('https://raw.githubusercontent.com/rogovich/Data/master/data/titanic/train.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Функция read_csv читает данные из файла формата csv и преобразует в pandas.DataFrame. Аналогичная функция read_excel может читать данные в офрмате xls(x).\n", "\n", "Посмотрим на наши данные:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass \\\n", "0 1 0 3 \n", "1 2 1 1 \n", "2 3 1 3 \n", "3 4 1 1 \n", "4 5 0 3 \n", "\n", " Name Sex Age SibSp \\\n", "0 Braund, Mr. Owen Harris male 22.0 1 \n", "1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 \n", "2 Heikkinen, Miss. Laina female 26.0 0 \n", "3 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 \n", "4 Allen, Mr. William Henry male 35.0 0 \n", "\n", " Parch Ticket Fare Cabin Embarked \n", "0 0 A/5 21171 7.2500 NaN S \n", "1 0 PC 17599 71.2833 C85 C \n", "2 0 STON/O2. 3101282 7.9250 NaN S \n", "3 0 113803 53.1000 C123 S \n", "4 0 373450 8.0500 NaN S " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.head() # функция head() показывает первые строки датафрейма, по умолчанию 5" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS
5603Moran, Mr. JamesmaleNaN003308778.4583NaNQ
6701McCarthy, Mr. Timothy Jmale54.0001746351.8625E46S
7803Palsson, Master. Gosta Leonardmale2.03134990921.0750NaNS
8913Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)female27.00234774211.1333NaNS
91012Nasser, Mrs. Nicholas (Adele Achem)female14.01023773630.0708NaNC
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass \\\n", "0 1 0 3 \n", "1 2 1 1 \n", "2 3 1 3 \n", "3 4 1 1 \n", "4 5 0 3 \n", "5 6 0 3 \n", "6 7 0 1 \n", "7 8 0 3 \n", "8 9 1 3 \n", "9 10 1 2 \n", "\n", " Name Sex Age SibSp \\\n", "0 Braund, Mr. Owen Harris male 22.0 1 \n", "1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 \n", "2 Heikkinen, Miss. Laina female 26.0 0 \n", "3 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 \n", "4 Allen, Mr. William Henry male 35.0 0 \n", "5 Moran, Mr. James male NaN 0 \n", "6 McCarthy, Mr. Timothy J male 54.0 0 \n", "7 Palsson, Master. Gosta Leonard male 2.0 3 \n", "8 Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg) female 27.0 0 \n", "9 Nasser, Mrs. Nicholas (Adele Achem) female 14.0 1 \n", "\n", " Parch Ticket Fare Cabin Embarked \n", "0 0 A/5 21171 7.2500 NaN S \n", "1 0 PC 17599 71.2833 C85 C \n", "2 0 STON/O2. 3101282 7.9250 NaN S \n", "3 0 113803 53.1000 C123 S \n", "4 0 373450 8.0500 NaN S \n", "5 0 330877 8.4583 NaN Q \n", "6 0 17463 51.8625 E46 S \n", "7 1 349909 21.0750 NaN S \n", "8 2 347742 11.1333 NaN S \n", "9 0 237736 30.0708 NaN C " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.head(10) # можно передать аргументом количество строк, которые хотите увидеть" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
88188203Markun, Mr. Johannmale33.0003492577.8958NaNS
88288303Dahlberg, Miss. Gerda Ulrikafemale22.000755210.5167NaNS
88388402Banfield, Mr. Frederick Jamesmale28.000C.A./SOTON 3406810.5000NaNS
88488503Sutehall, Mr. Henry Jrmale25.000SOTON/OQ 3920767.0500NaNS
88588603Rice, Mrs. William (Margaret Norton)female39.00538265229.1250NaNQ
88688702Montvila, Rev. Juozasmale27.00021153613.0000NaNS
88788811Graham, Miss. Margaret Edithfemale19.00011205330.0000B42S
88888903Johnston, Miss. Catherine Helen \"Carrie\"femaleNaN12W./C. 660723.4500NaNS
88989011Behr, Mr. Karl Howellmale26.00011136930.0000C148C
89089103Dooley, Mr. Patrickmale32.0003703767.7500NaNQ
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass Name \\\n", "881 882 0 3 Markun, Mr. Johann \n", "882 883 0 3 Dahlberg, Miss. Gerda Ulrika \n", "883 884 0 2 Banfield, Mr. Frederick James \n", "884 885 0 3 Sutehall, Mr. Henry Jr \n", "885 886 0 3 Rice, Mrs. William (Margaret Norton) \n", "886 887 0 2 Montvila, Rev. Juozas \n", "887 888 1 1 Graham, Miss. Margaret Edith \n", "888 889 0 3 Johnston, Miss. Catherine Helen \"Carrie\" \n", "889 890 1 1 Behr, Mr. Karl Howell \n", "890 891 0 3 Dooley, Mr. Patrick \n", "\n", " Sex Age SibSp Parch Ticket Fare Cabin Embarked \n", "881 male 33.0 0 0 349257 7.8958 NaN S \n", "882 female 22.0 0 0 7552 10.5167 NaN S \n", "883 male 28.0 0 0 C.A./SOTON 34068 10.5000 NaN S \n", "884 male 25.0 0 0 SOTON/OQ 392076 7.0500 NaN S \n", "885 female 39.0 0 5 382652 29.1250 NaN Q \n", "886 male 27.0 0 0 211536 13.0000 NaN S \n", "887 female 19.0 0 0 112053 30.0000 B42 S \n", "888 female NaN 1 2 W./C. 6607 23.4500 NaN S \n", "889 male 26.0 0 0 111369 30.0000 C148 C \n", "890 male 32.0 0 0 370376 7.7500 NaN Q " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.tail(10) # можно посмотреть последние записи" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "По столбцам идут признаки, по строкам - объекты (пассажиры)." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(891, 12)" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.shape # функция shape показывает размерность датафрейма (строк, столбцов)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Index(['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp',\n", " 'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked'],\n", " dtype='object')" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.columns # список столбцов " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Описание признаков:\n", "\n", "**PassengerId** - id пассажира\n", "\n", "**Survived** бинарная переменная: выжил пассажирил (1) или нет (0)\n", "\n", "**Pclass** - класс пассажира\n", "\n", "**Name** - имя пассажира\n", "\n", "**Sex** - пол пассажира\n", "\n", "**Age** - возраст пассажира\n", "\n", "**SibSp** - количество родственников (братьев, сестер, супругов) пассажира на борту\n", "\n", "**Parch** - количество родственников (родителей / детей) пассажира на борту\n", "\n", "**Ticket** - номер билета\n", "\n", "**Fare** - тариф (стоимость билета)\n", "\n", "**Cabin** - номер кабины\n", "\n", "**Embarked** - порт, в котором пассажир сел на борт (C - Cherbourg, S - Southampton, Q = Queenstown)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Так можно обратиться к столбцу:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 22.0\n", "1 38.0\n", "2 26.0\n", "3 35.0\n", "4 35.0\n", "Name: Age, dtype: float64" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['Age'].head()" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 22.0\n", "1 38.0\n", "2 26.0\n", "3 35.0\n", "4 35.0\n", "Name: Age, dtype: float64" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.Age.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Или к нескольким столбцам сразу:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
AgeSex
022.0male
138.0female
226.0female
335.0female
435.0male
\n", "
" ], "text/plain": [ " Age Sex\n", "0 22.0 male\n", "1 38.0 female\n", "2 26.0 female\n", "3 35.0 female\n", "4 35.0 male" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data[['Age', 'Sex']].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A так - к строке по индексу:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "PassengerId 1\n", "Survived 0\n", "Pclass 3\n", "Name Braund, Mr. Owen Harris\n", "Sex male\n", "Age 22.0\n", "SibSp 1\n", "Parch 0\n", "Ticket A/5 21171\n", "Fare 7.25\n", "Cabin NaN\n", "Embarked S\n", "Name: 0, dtype: object" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.loc[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Кроме того, можно выбирать объекты, удовлетворяющие каким-то свойствам, например, все пассажиры-женщины:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
8913Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)female27.00234774211.1333NaNS
91012Nasser, Mrs. Nicholas (Adele Achem)female14.01023773630.0708NaNC
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass \\\n", "1 2 1 1 \n", "2 3 1 3 \n", "3 4 1 1 \n", "8 9 1 3 \n", "9 10 1 2 \n", "\n", " Name Sex Age SibSp \\\n", "1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 \n", "2 Heikkinen, Miss. Laina female 26.0 0 \n", "3 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 \n", "8 Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg) female 27.0 0 \n", "9 Nasser, Mrs. Nicholas (Adele Achem) female 14.0 1 \n", "\n", " Parch Ticket Fare Cabin Embarked \n", "1 0 PC 17599 71.2833 C85 C \n", "2 0 STON/O2. 3101282 7.9250 NaN S \n", "3 0 113803 53.1000 C123 S \n", "8 2 347742 11.1333 NaN S \n", "9 0 237736 30.0708 NaN C " ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data[data.Sex == 'female'].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Пассажиры первого класса:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
6701McCarthy, Mr. Timothy Jmale54.0001746351.8625E46S
111211Bonnell, Miss. Elizabethfemale58.00011378326.5500C103S
232411Sloper, Mr. William Thompsonmale28.00011378835.5000A6S
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass \\\n", "1 2 1 1 \n", "3 4 1 1 \n", "6 7 0 1 \n", "11 12 1 1 \n", "23 24 1 1 \n", "\n", " Name Sex Age SibSp \\\n", "1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 \n", "3 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 \n", "6 McCarthy, Mr. Timothy J male 54.0 0 \n", "11 Bonnell, Miss. Elizabeth female 58.0 0 \n", "23 Sloper, Mr. William Thompson male 28.0 0 \n", "\n", " Parch Ticket Fare Cabin Embarked \n", "1 0 PC 17599 71.2833 C85 C \n", "3 0 113803 53.1000 C123 S \n", "6 0 17463 51.8625 E46 S \n", "11 0 113783 26.5500 C103 S \n", "23 0 113788 35.5000 A6 S " ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data[data.Pclass == 1].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Пассажиры первого или второго классов:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
6701McCarthy, Mr. Timothy Jmale54.0001746351.8625E46S
91012Nasser, Mrs. Nicholas (Adele Achem)female14.01023773630.0708NaNC
111211Bonnell, Miss. Elizabethfemale58.00011378326.5500C103S
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass \\\n", "1 2 1 1 \n", "3 4 1 1 \n", "6 7 0 1 \n", "9 10 1 2 \n", "11 12 1 1 \n", "\n", " Name Sex Age SibSp \\\n", "1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 \n", "3 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 \n", "6 McCarthy, Mr. Timothy J male 54.0 0 \n", "9 Nasser, Mrs. Nicholas (Adele Achem) female 14.0 1 \n", "11 Bonnell, Miss. Elizabeth female 58.0 0 \n", "\n", " Parch Ticket Fare Cabin Embarked \n", "1 0 PC 17599 71.2833 C85 C \n", "3 0 113803 53.1000 C123 S \n", "6 0 17463 51.8625 E46 S \n", "9 0 237736 30.0708 NaN C \n", "11 0 113783 26.5500 C103 S " ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data[data.Pclass.isin([1,2])].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Пассажиры младше 18:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
7803Palsson, Master. Gosta Leonardmale2.03134990921.0750NaNS
91012Nasser, Mrs. Nicholas (Adele Achem)female14.01023773630.0708NaNC
101113Sandstrom, Miss. Marguerite Rutfemale4.011PP 954916.7000G6S
141503Vestrom, Miss. Hulda Amanda Adolfinafemale14.0003504067.8542NaNS
161703Rice, Master. Eugenemale2.04138265229.1250NaNQ
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass Name \\\n", "7 8 0 3 Palsson, Master. Gosta Leonard \n", "9 10 1 2 Nasser, Mrs. Nicholas (Adele Achem) \n", "10 11 1 3 Sandstrom, Miss. Marguerite Rut \n", "14 15 0 3 Vestrom, Miss. Hulda Amanda Adolfina \n", "16 17 0 3 Rice, Master. Eugene \n", "\n", " Sex Age SibSp Parch Ticket Fare Cabin Embarked \n", "7 male 2.0 3 1 349909 21.0750 NaN S \n", "9 female 14.0 1 0 237736 30.0708 NaN C \n", "10 female 4.0 1 1 PP 9549 16.7000 G6 S \n", "14 female 14.0 0 0 350406 7.8542 NaN S \n", "16 male 2.0 4 1 382652 29.1250 NaN Q " ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data[data.Age < 18].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Девушки в возрасте от 18 до 25, путешествующие в одиночку (без каких-либо родственников):" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
444513Devaney, Miss. Margaret Deliafemale19.0003309587.8792NaNQ
565712Rugg, Miss. Emilyfemale21.000C.A. 3102610.5000NaNS
10610713Salkjelsvik, Miss. Anna Kristinefemale21.0003431207.6500NaNS
14114213Nysten, Miss. Anna Sofiafemale22.0003470817.7500NaNS
19920002Yrois, Miss. Henriette (\"Mrs Harbeck\")female24.00024874713.0000NaNS
28929013Connolly, Miss. Katefemale22.0003703737.7500NaNQ
29329403Haas, Miss. Aloisiafemale24.0003492368.8500NaNS
31031111Hays, Miss. Margaret Bechsteinfemale24.0001176783.1583C54C
34534612Brown, Miss. Amelia \"Mildred\"female24.00024873313.0000F33S
36937011Aubart, Mme. Leontine Paulinefemale24.000PC 1747769.3000B35C
37637713Landergren, Miss. Aurora Adeliafemale22.000C 70777.2500NaNS
40440503Oreskovic, Miss. Marijafemale20.0003150968.6625NaNS
42742812Phillips, Miss. Kate Florence (\"Mrs Kate Louis...female19.00025065526.0000NaNS
47347412Jerwan, Mrs. Amin S (Marie Marthe Thuillard)female23.000SC/AH Basle 54113.7917DC
47447503Strandberg, Miss. Ida Sofiafemale22.00075539.8375NaNS
50150203Canavan, Miss. Maryfemale21.0003648467.7500NaNQ
55455513Ohman, Miss. Velinfemale22.0003470857.7750NaNS
62762811Longley, Miss. Gretchen Fiskefemale21.0001350277.9583D9S
64164211Sagesser, Mlle. Emmafemale24.000PC 1747769.3000B35C
64965013Stanley, Miss. Amy Zillah Elsiefemale23.000CA. 23147.5500NaNS
70870911Cleaver, Miss. Alicefemale22.000113781151.5500NaNS
71071111Mayne, Mlle. Berthe Antonine (\"Mrs de Villiers\")female24.000PC 1748249.5042C90C
81681703Heininen, Miss. Wendla Mariafemale23.000STON/O2. 31012907.9250NaNS
88288303Dahlberg, Miss. Gerda Ulrikafemale22.000755210.5167NaNS
88788811Graham, Miss. Margaret Edithfemale19.00011205330.0000B42S
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass \\\n", "44 45 1 3 \n", "56 57 1 2 \n", "106 107 1 3 \n", "141 142 1 3 \n", "199 200 0 2 \n", "289 290 1 3 \n", "293 294 0 3 \n", "310 311 1 1 \n", "345 346 1 2 \n", "369 370 1 1 \n", "376 377 1 3 \n", "404 405 0 3 \n", "427 428 1 2 \n", "473 474 1 2 \n", "474 475 0 3 \n", "501 502 0 3 \n", "554 555 1 3 \n", "627 628 1 1 \n", "641 642 1 1 \n", "649 650 1 3 \n", "708 709 1 1 \n", "710 711 1 1 \n", "816 817 0 3 \n", "882 883 0 3 \n", "887 888 1 1 \n", "\n", " Name Sex Age SibSp \\\n", "44 Devaney, Miss. Margaret Delia female 19.0 0 \n", "56 Rugg, Miss. Emily female 21.0 0 \n", "106 Salkjelsvik, Miss. Anna Kristine female 21.0 0 \n", "141 Nysten, Miss. Anna Sofia female 22.0 0 \n", "199 Yrois, Miss. Henriette (\"Mrs Harbeck\") female 24.0 0 \n", "289 Connolly, Miss. Kate female 22.0 0 \n", "293 Haas, Miss. Aloisia female 24.0 0 \n", "310 Hays, Miss. Margaret Bechstein female 24.0 0 \n", "345 Brown, Miss. Amelia \"Mildred\" female 24.0 0 \n", "369 Aubart, Mme. Leontine Pauline female 24.0 0 \n", "376 Landergren, Miss. Aurora Adelia female 22.0 0 \n", "404 Oreskovic, Miss. Marija female 20.0 0 \n", "427 Phillips, Miss. Kate Florence (\"Mrs Kate Louis... female 19.0 0 \n", "473 Jerwan, Mrs. Amin S (Marie Marthe Thuillard) female 23.0 0 \n", "474 Strandberg, Miss. Ida Sofia female 22.0 0 \n", "501 Canavan, Miss. Mary female 21.0 0 \n", "554 Ohman, Miss. Velin female 22.0 0 \n", "627 Longley, Miss. Gretchen Fiske female 21.0 0 \n", "641 Sagesser, Mlle. Emma female 24.0 0 \n", "649 Stanley, Miss. Amy Zillah Elsie female 23.0 0 \n", "708 Cleaver, Miss. Alice female 22.0 0 \n", "710 Mayne, Mlle. Berthe Antonine (\"Mrs de Villiers\") female 24.0 0 \n", "816 Heininen, Miss. Wendla Maria female 23.0 0 \n", "882 Dahlberg, Miss. Gerda Ulrika female 22.0 0 \n", "887 Graham, Miss. Margaret Edith female 19.0 0 \n", "\n", " Parch Ticket Fare Cabin Embarked \n", "44 0 330958 7.8792 NaN Q \n", "56 0 C.A. 31026 10.5000 NaN S \n", "106 0 343120 7.6500 NaN S \n", "141 0 347081 7.7500 NaN S \n", "199 0 248747 13.0000 NaN S \n", "289 0 370373 7.7500 NaN Q \n", "293 0 349236 8.8500 NaN S \n", "310 0 11767 83.1583 C54 C \n", "345 0 248733 13.0000 F33 S \n", "369 0 PC 17477 69.3000 B35 C \n", "376 0 C 7077 7.2500 NaN S \n", "404 0 315096 8.6625 NaN S \n", "427 0 250655 26.0000 NaN S \n", "473 0 SC/AH Basle 541 13.7917 D C \n", "474 0 7553 9.8375 NaN S \n", "501 0 364846 7.7500 NaN Q \n", "554 0 347085 7.7750 NaN S \n", "627 0 13502 77.9583 D9 S \n", "641 0 PC 17477 69.3000 B35 C \n", "649 0 CA. 2314 7.5500 NaN S \n", "708 0 113781 151.5500 NaN S \n", "710 0 PC 17482 49.5042 C90 C \n", "816 0 STON/O2. 3101290 7.9250 NaN S \n", "882 0 7552 10.5167 NaN S \n", "887 0 112053 30.0000 B42 S " ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data[(data.Sex == 'female') & (data.Age > 18) & (data.Age < 25) & (data.SibSp == 0) &(data.Parch == 0)]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Найдем женщин от 18 до 25 без родственников. Сколько таких путещественниц?" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(25, 12)" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data[(data.Sex == 'female') & (data.Age > 18) & (data.Age < 25) & (data.SibSp == 0) &(data.Parch == 0)].shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Иногда нужно создать новый признак из уже существующих, например, нам интересно, сколько всего родственников путешествовало с каждым пассажиром - просто сложим столбцы SibSp и Parch и поместим сумму в новый столбец FamilySize." ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarkedFamilySize
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS1
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C1
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS0
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S1
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS0
5603Moran, Mr. JamesmaleNaN003308778.4583NaNQ0
6701McCarthy, Mr. Timothy Jmale54.0001746351.8625E46S0
7803Palsson, Master. Gosta Leonardmale2.03134990921.0750NaNS4
8913Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)female27.00234774211.1333NaNS2
91012Nasser, Mrs. Nicholas (Adele Achem)female14.01023773630.0708NaNC1
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass \\\n", "0 1 0 3 \n", "1 2 1 1 \n", "2 3 1 3 \n", "3 4 1 1 \n", "4 5 0 3 \n", "5 6 0 3 \n", "6 7 0 1 \n", "7 8 0 3 \n", "8 9 1 3 \n", "9 10 1 2 \n", "\n", " Name Sex Age SibSp \\\n", "0 Braund, Mr. Owen Harris male 22.0 1 \n", "1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 \n", "2 Heikkinen, Miss. Laina female 26.0 0 \n", "3 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 \n", "4 Allen, Mr. William Henry male 35.0 0 \n", "5 Moran, Mr. James male NaN 0 \n", "6 McCarthy, Mr. Timothy J male 54.0 0 \n", "7 Palsson, Master. Gosta Leonard male 2.0 3 \n", "8 Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg) female 27.0 0 \n", "9 Nasser, Mrs. Nicholas (Adele Achem) female 14.0 1 \n", "\n", " Parch Ticket Fare Cabin Embarked FamilySize \n", "0 0 A/5 21171 7.2500 NaN S 1 \n", "1 0 PC 17599 71.2833 C85 C 1 \n", "2 0 STON/O2. 3101282 7.9250 NaN S 0 \n", "3 0 113803 53.1000 C123 S 1 \n", "4 0 373450 8.0500 NaN S 0 \n", "5 0 330877 8.4583 NaN Q 0 \n", "6 0 17463 51.8625 E46 S 0 \n", "7 1 349909 21.0750 NaN S 4 \n", "8 2 347742 11.1333 NaN S 2 \n", "9 0 237736 30.0708 NaN C 1 " ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['FamilySize'] = data['SibSp'] + data['Parch']\n", "data.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "А теперь давайте создадим переменную, которая бы нам показывала, что пассажир ехал в одиночку. Такой пассажир путешествовал без родственников. Мы напишем условие с помощью анонимной функции (1, если FamilySize равно 0 и 0 во всех остальных случаях) и применим ее к столбцу FamilySize с помощью метода .apply()." ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarkedFamilySizeAlone
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS10
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C10
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS01
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S10
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS01
5603Moran, Mr. JamesmaleNaN003308778.4583NaNQ01
6701McCarthy, Mr. Timothy Jmale54.0001746351.8625E46S01
7803Palsson, Master. Gosta Leonardmale2.03134990921.0750NaNS40
8913Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)female27.00234774211.1333NaNS20
91012Nasser, Mrs. Nicholas (Adele Achem)female14.01023773630.0708NaNC10
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass \\\n", "0 1 0 3 \n", "1 2 1 1 \n", "2 3 1 3 \n", "3 4 1 1 \n", "4 5 0 3 \n", "5 6 0 3 \n", "6 7 0 1 \n", "7 8 0 3 \n", "8 9 1 3 \n", "9 10 1 2 \n", "\n", " Name Sex Age SibSp \\\n", "0 Braund, Mr. Owen Harris male 22.0 1 \n", "1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 \n", "2 Heikkinen, Miss. Laina female 26.0 0 \n", "3 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 \n", "4 Allen, Mr. William Henry male 35.0 0 \n", "5 Moran, Mr. James male NaN 0 \n", "6 McCarthy, Mr. Timothy J male 54.0 0 \n", "7 Palsson, Master. Gosta Leonard male 2.0 3 \n", "8 Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg) female 27.0 0 \n", "9 Nasser, Mrs. Nicholas (Adele Achem) female 14.0 1 \n", "\n", " Parch Ticket Fare Cabin Embarked FamilySize Alone \n", "0 0 A/5 21171 7.2500 NaN S 1 0 \n", "1 0 PC 17599 71.2833 C85 C 1 0 \n", "2 0 STON/O2. 3101282 7.9250 NaN S 0 1 \n", "3 0 113803 53.1000 C123 S 1 0 \n", "4 0 373450 8.0500 NaN S 0 1 \n", "5 0 330877 8.4583 NaN Q 0 1 \n", "6 0 17463 51.8625 E46 S 0 1 \n", "7 1 349909 21.0750 NaN S 4 0 \n", "8 2 347742 11.1333 NaN S 2 0 \n", "9 0 237736 30.0708 NaN C 1 0 " ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['Alone'] = data['FamilySize'].apply(lambda x: 1 if x == 0 else 0)\n", "data.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Функция, которую применяем к столбцу, может быть и посложнее. Давайте из каждого имени пассажира достанем его титул. Сначала потренируемся на одном пассажире." ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Braund, Mr. Owen Harris'" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.loc[0]['Name']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ок, выбрали имя. Это строка. Давайте подумаем, как достать из нее титул. Вроде бы титул всегда после фамилии. Разобьем строку по запятой и достанем второй (первый по индексу) элемент. Потом разобьем по точке и возьмем первый (нулевой) элемент. И избавимся от пробела." ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Mr'" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.loc[0]['Name'].split(',')[1].split('.')[0].strip()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ура! Теперь напишем функцию, которая будет забирать титул из имени, а потом применим ее к колонке Name." ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [], "source": [ "def return_title(full_name):\n", " return(full_name.split(',')[1].split('.')[0].strip())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Теперь сформируем новый столбец family_name из столбца Name с помощью написанной нами функции:" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarkedFamilySizeAloneTitle
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS10Mr
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C10Mrs
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS01Miss
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S10Mrs
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS01Mr
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass \\\n", "0 1 0 3 \n", "1 2 1 1 \n", "2 3 1 3 \n", "3 4 1 1 \n", "4 5 0 3 \n", "\n", " Name Sex Age SibSp \\\n", "0 Braund, Mr. Owen Harris male 22.0 1 \n", "1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 \n", "2 Heikkinen, Miss. Laina female 26.0 0 \n", "3 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 \n", "4 Allen, Mr. William Henry male 35.0 0 \n", "\n", " Parch Ticket Fare Cabin Embarked FamilySize Alone Title \n", "0 0 A/5 21171 7.2500 NaN S 1 0 Mr \n", "1 0 PC 17599 71.2833 C85 C 1 0 Mrs \n", "2 0 STON/O2. 3101282 7.9250 NaN S 0 1 Miss \n", "3 0 113803 53.1000 C123 S 1 0 Mrs \n", "4 0 373450 8.0500 NaN S 0 1 Mr " ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['Title'] = data.Name.apply(return_title)\n", "data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Полезно посмотреть, какие значения содержатся в столбце. Для категориальных данных можно посчитать частоту встречаемости каждого значения с помошью функции value_counts:" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Mr 517\n", "Miss 182\n", "Mrs 125\n", "Master 40\n", "Dr 7\n", "Rev 6\n", "Mlle 2\n", "Major 2\n", "Col 2\n", "the Countess 1\n", "Capt 1\n", "Ms 1\n", "Sir 1\n", "Lady 1\n", "Mme 1\n", "Don 1\n", "Jonkheer 1\n", "Name: Title, dtype: int64" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['Title'].value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Очень много уникальных значений! Обычно это не очень хорошо для статистического анализа, давайте все такие титулы переиминуем в Misc (другие)." ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [], "source": [ "data['Title'] = data['Title'].apply(lambda x: 'Misc' if x not in ['Mr', 'Miss', 'Mrs', 'Master'] else x)" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Mr 517\n", "Miss 182\n", "Mrs 125\n", "Master 40\n", "Misc 27\n", "Name: Title, dtype: int64" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['Title'].value_counts()" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "3 491\n", "1 216\n", "2 184\n", "Name: Pclass, dtype: int64" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['Pclass'].value_counts()" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "S 644\n", "C 168\n", "Q 77\n", "Name: Embarked, dtype: int64" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['Embarked'].value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Для количественных данных удобнее смотреть минимальные/максимальные/средние значения:" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.42\n", "80.0\n", "29.69911764705882\n" ] } ], "source": [ "print(data['Age'].min()) \n", "print(data['Age'].max()) \n", "print(data['Age'].mean())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "В Pandas есть функция describe(), которая делает удобную сводную таблицу по всем количественным столбцам сразу (обратите внимание, что для Pandas количественные данные = все, что представлено числами, что, разумеется, неверно в общем случае):" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassAgeSibSpParchFareFamilySizeAlone
count891.000000891.000000891.000000714.000000891.000000891.000000891.000000891.000000891.000000
mean446.0000000.3838382.30864229.6991180.5230080.38159432.2042080.9046020.602694
std257.3538420.4865920.83607114.5264971.1027430.80605749.6934291.6134590.489615
min1.0000000.0000001.0000000.4200000.0000000.0000000.0000000.0000000.000000
25%223.5000000.0000002.00000020.1250000.0000000.0000007.9104000.0000000.000000
50%446.0000000.0000003.00000028.0000000.0000000.00000014.4542000.0000001.000000
75%668.5000001.0000003.00000038.0000001.0000000.00000031.0000001.0000001.000000
max891.0000001.0000003.00000080.0000008.0000006.000000512.32920010.0000001.000000
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass Age SibSp \\\n", "count 891.000000 891.000000 891.000000 714.000000 891.000000 \n", "mean 446.000000 0.383838 2.308642 29.699118 0.523008 \n", "std 257.353842 0.486592 0.836071 14.526497 1.102743 \n", "min 1.000000 0.000000 1.000000 0.420000 0.000000 \n", "25% 223.500000 0.000000 2.000000 20.125000 0.000000 \n", "50% 446.000000 0.000000 3.000000 28.000000 0.000000 \n", "75% 668.500000 1.000000 3.000000 38.000000 1.000000 \n", "max 891.000000 1.000000 3.000000 80.000000 8.000000 \n", "\n", " Parch Fare FamilySize Alone \n", "count 891.000000 891.000000 891.000000 891.000000 \n", "mean 0.381594 32.204208 0.904602 0.602694 \n", "std 0.806057 49.693429 1.613459 0.489615 \n", "min 0.000000 0.000000 0.000000 0.000000 \n", "25% 0.000000 7.910400 0.000000 0.000000 \n", "50% 0.000000 14.454200 0.000000 1.000000 \n", "75% 0.000000 31.000000 1.000000 1.000000 \n", "max 6.000000 512.329200 10.000000 1.000000 " ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.describe()" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
AgeFare
count714.000000891.000000
mean29.69911832.204208
std14.52649749.693429
min0.4200000.000000
25%20.1250007.910400
50%28.00000014.454200
75%38.00000031.000000
max80.000000512.329200
\n", "
" ], "text/plain": [ " Age Fare\n", "count 714.000000 891.000000\n", "mean 29.699118 32.204208\n", "std 14.526497 49.693429\n", "min 0.420000 0.000000\n", "25% 20.125000 7.910400\n", "50% 28.000000 14.454200\n", "75% 38.000000 31.000000\n", "max 80.000000 512.329200" ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data[['Age', 'Fare']].describe() # также можно применять только к отдельным колонкам" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Данные можно сортировать:" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [], "source": [ "data = data.dropna(how = 'any')" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Int64Index: 183 entries, 1 to 889\n", "Data columns (total 15 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 PassengerId 183 non-null int64 \n", " 1 Survived 183 non-null int64 \n", " 2 Pclass 183 non-null int64 \n", " 3 Name 183 non-null object \n", " 4 Sex 183 non-null object \n", " 5 Age 183 non-null float64\n", " 6 SibSp 183 non-null int64 \n", " 7 Parch 183 non-null int64 \n", " 8 Ticket 183 non-null object \n", " 9 Fare 183 non-null float64\n", " 10 Cabin 183 non-null object \n", " 11 Embarked 183 non-null object \n", " 12 FamilySize 183 non-null int64 \n", " 13 Alone 183 non-null int64 \n", " 14 Title 183 non-null object \n", "dtypes: float64(2), int64(7), object(6)\n", "memory usage: 22.9+ KB\n" ] } ], "source": [ "data.info()" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarkedFamilySizeAloneTitle
545501Ostby, Mr. Engelhart Corneliusmale65.00111350961.9792B30C10Mr
45645701Millet, Mr. Francis Davismale65.0001350926.5500E38S01Mr
74574601Crosby, Capt. Edward Giffordmale70.011WE/P 573571.0000B22S20Misc
969701Goldschmidt, Mr. George Bmale71.000PC 1775434.6542A5C01Mr
63063111Barkworth, Mr. Algernon Henry Wilsonmale80.0002704230.0000A23S01Mr
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass Name \\\n", "54 55 0 1 Ostby, Mr. Engelhart Cornelius \n", "456 457 0 1 Millet, Mr. Francis Davis \n", "745 746 0 1 Crosby, Capt. Edward Gifford \n", "96 97 0 1 Goldschmidt, Mr. George B \n", "630 631 1 1 Barkworth, Mr. Algernon Henry Wilson \n", "\n", " Sex Age SibSp Parch Ticket Fare Cabin Embarked FamilySize \\\n", "54 male 65.0 0 1 113509 61.9792 B30 C 1 \n", "456 male 65.0 0 0 13509 26.5500 E38 S 0 \n", "745 male 70.0 1 1 WE/P 5735 71.0000 B22 S 2 \n", "96 male 71.0 0 0 PC 17754 34.6542 A5 C 0 \n", "630 male 80.0 0 0 27042 30.0000 A23 S 0 \n", "\n", " Alone Title \n", "54 0 Mr \n", "456 1 Mr \n", "745 0 Misc \n", "96 1 Mr \n", "630 1 Mr " ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.sort_values(by=['Age']).tail() # сортируем по возрасту, по умолчанию сортирвка по возрастанию" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarkedFamilySizeAloneTitle
63063111Barkworth, Mr. Algernon Henry Wilsonmale80.0002704230.0000A23S01Mr
969701Goldschmidt, Mr. George Bmale71.000PC 1775434.6542A5C01Mr
74574601Crosby, Capt. Edward Giffordmale70.011WE/P 573571.0000B22S20Misc
45645701Millet, Mr. Francis Davismale65.0001350926.5500E38S01Mr
545501Ostby, Mr. Engelhart Corneliusmale65.00111350961.9792B30C10Mr
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass Name \\\n", "630 631 1 1 Barkworth, Mr. Algernon Henry Wilson \n", "96 97 0 1 Goldschmidt, Mr. George B \n", "745 746 0 1 Crosby, Capt. Edward Gifford \n", "456 457 0 1 Millet, Mr. Francis Davis \n", "54 55 0 1 Ostby, Mr. Engelhart Cornelius \n", "\n", " Sex Age SibSp Parch Ticket Fare Cabin Embarked FamilySize \\\n", "630 male 80.0 0 0 27042 30.0000 A23 S 0 \n", "96 male 71.0 0 0 PC 17754 34.6542 A5 C 0 \n", "745 male 70.0 1 1 WE/P 5735 71.0000 B22 S 2 \n", "456 male 65.0 0 0 13509 26.5500 E38 S 0 \n", "54 male 65.0 0 1 113509 61.9792 B30 C 1 \n", "\n", " Alone Title \n", "630 1 Mr \n", "96 1 Mr \n", "745 0 Misc \n", "456 1 Mr \n", "54 0 Mr " ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.sort_values(by=['Age'], ascending=False).head() # сортируем по возрасту, теперь по убыванию" ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarkedFamilySizeAloneTitle
63063111Barkworth, Mr. Algernon Henry Wilsonmale80.0002704230.0000A23S01Mr
969701Goldschmidt, Mr. George Bmale71.000PC 1775434.6542A5C01Mr
74574601Crosby, Capt. Edward Giffordmale70.011WE/P 573571.0000B22S20Misc
545501Ostby, Mr. Engelhart Corneliusmale65.00111350961.9792B30C10Mr
45645701Millet, Mr. Francis Davismale65.0001350926.5500E38S01Mr
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass Name \\\n", "630 631 1 1 Barkworth, Mr. Algernon Henry Wilson \n", "96 97 0 1 Goldschmidt, Mr. George B \n", "745 746 0 1 Crosby, Capt. Edward Gifford \n", "54 55 0 1 Ostby, Mr. Engelhart Cornelius \n", "456 457 0 1 Millet, Mr. Francis Davis \n", "\n", " Sex Age SibSp Parch Ticket Fare Cabin Embarked FamilySize \\\n", "630 male 80.0 0 0 27042 30.0000 A23 S 0 \n", "96 male 71.0 0 0 PC 17754 34.6542 A5 C 0 \n", "745 male 70.0 1 1 WE/P 5735 71.0000 B22 S 2 \n", "54 male 65.0 0 1 113509 61.9792 B30 C 1 \n", "456 male 65.0 0 0 13509 26.5500 E38 S 0 \n", "\n", " Alone Title \n", "630 1 Mr \n", "96 1 Mr \n", "745 0 Misc \n", "54 0 Mr \n", "456 1 Mr " ] }, "execution_count": 48, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.sort_values(by=['Age', 'Fare'], ascending=False).head() # сортируем сперва по возрасту (по убыванию),\n", " # потом стоимости билета (по убыванию)" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarkedFamilySizeAloneTitle
63063111Barkworth, Mr. Algernon Henry Wilsonmale80.0002704230.0000A23S01Mr
969701Goldschmidt, Mr. George Bmale71.000PC 1775434.6542A5C01Mr
74574601Crosby, Capt. Edward Giffordmale70.011WE/P 573571.0000B22S20Misc
45645701Millet, Mr. Francis Davismale65.0001350926.5500E38S01Mr
545501Ostby, Mr. Engelhart Corneliusmale65.00111350961.9792B30C10Mr
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass Name \\\n", "630 631 1 1 Barkworth, Mr. Algernon Henry Wilson \n", "96 97 0 1 Goldschmidt, Mr. George B \n", "745 746 0 1 Crosby, Capt. Edward Gifford \n", "456 457 0 1 Millet, Mr. Francis Davis \n", "54 55 0 1 Ostby, Mr. Engelhart Cornelius \n", "\n", " Sex Age SibSp Parch Ticket Fare Cabin Embarked FamilySize \\\n", "630 male 80.0 0 0 27042 30.0000 A23 S 0 \n", "96 male 71.0 0 0 PC 17754 34.6542 A5 C 0 \n", "745 male 70.0 1 1 WE/P 5735 71.0000 B22 S 2 \n", "456 male 65.0 0 0 13509 26.5500 E38 S 0 \n", "54 male 65.0 0 1 113509 61.9792 B30 C 1 \n", "\n", " Alone Title \n", "630 1 Mr \n", "96 1 Mr \n", "745 0 Misc \n", "456 1 Mr \n", "54 0 Mr " ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.sort_values(by=['Age', 'Fare'], ascending=[False, True]).head() # сортируем сперва по возрасту (по убыванию),\n", " # потом стоимости билета (по возрастанию)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Сохранение датафрейма:" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [], "source": [ "data.to_csv('new_titanic.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Описательные статистики в Python\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Максимальное и минимальное значения**" ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.92" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.Age.min()" ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "80.0" ] }, "execution_count": 52, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.Age.max()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Меры центральной тенденции**\n", "\n", "Меры среднего уровня дают усредненную характеристику совокупности объектов по определенному признаку.\n", "\n", "В зависимости от типа шкалы измерения применяются следующие меры среднего уровня:\n", "- для количественных данных - среднее значение (арифметическое среднее), медиана, мода\n", "\n", "- для категориальных:\n", "\n", " для порядковых данных - медиана, мода\n", "\n", " для номинальных данных - мода\n", "\n", "На самом деле таких мер довольно много, например: взвешенное среднее — среднее значение, учитывающее весовые коэффициенты для каждого значения, гармоническое среднее — количество наблюдений, деленное на сумму инвертированных значений наблюдений, и так далее." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Среднее значение (математическое ожидание)**\n", "\n", "*Математическое ожидание* вычисляется по формуле:\n", "\n", "$$\n", " {\\displaystyle {\\bar {x}}={\\frac {1}{n}}\\sum _{i=1}^{n}x_{i}={\\frac {1}{n}}(x_{1}+\\ldots +x_{n})} \n", " $$" ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "35.6744262295082" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.Age.mean()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Немного усложним задачу**" ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Age
Pclass
137.591266
225.266667
321.000000
\n", "
" ], "text/plain": [ " Age\n", "Pclass \n", "1 37.591266\n", "2 25.266667\n", "3 21.000000" ] }, "execution_count": 54, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data[['Pclass', 'Age']].groupby('Pclass').mean()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Медиана**\n", "\n", "Если $x_1, x_2, ..., x_n$ – упорядоченные по возрастанию или убыванию числовые значения рассматриваемого признака, \n", "$n$ – объем выборки, то *медиана* - это средний элемент для нечетного $n$ и полуcумма средних элементов для четного $n$.\n", "\n", "Для порядковой шкалы медиана является такой точкой на шкале, которая делит всю совокупность опрошенных на две равных части — тех, кто отметил градации меньше этой точки (либо равные ей), и тех, кто отметил градации больше этой точки.\n", "\n", "Вопрос знатокам: можно ли посчитать медиану для категориальных не порядковых данных? 🤔" ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "36.0" ] }, "execution_count": 55, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.Age.median()" ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [ { "ename": "SyntaxError", "evalue": "invalid syntax (2413915786.py, line 1)", "output_type": "error", "traceback": [ "\u001b[1;36m File \u001b[1;32m\"C:\\Users\\mbbur\\AppData\\Local\\Temp\\ipykernel_35124\\2413915786.py\"\u001b[1;36m, line \u001b[1;32m1\u001b[0m\n\u001b[1;33m median_fare = #?\u001b[0m\n\u001b[1;37m ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n" ] } ], "source": [ "median_fare = #?\n", "print(\"Медиана стоимости билета на Титанике составляла %s\" % (median_fare))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Мода**\n", "\n", "Мода - значение во множестве наблюдений, которое встречается наиболее часто." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data.Pclass.mode()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data.Pclass.value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Меры разброса**\n", "\n", "Меры разброса показывают, насколько хорошо данные значения представляют совокупность. Как меры разброса применяются:\n", "\n", "- дисперсия случайной величины и среднеквадратическое отклонение\n", "\n", "- коэффициент вариации (это отношение среднеквадратического отклонения к среднему значению, выраженное в процентах, показывает однородность выборки)\n", "\n", "- и так далее" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Дисперсия и среднеквадратическое отклонение**\n", "\n", "*Дисперсия* значений признака является суммой квадратов отклонений этих значений от их среднего, деленной на число наблюдений:\n", "$$\n", "\\sigma^2={\\dfrac {\\sum \\limits _{i=1}^{n}\\left(x_{i}-{\\bar {x}}\\right)}{n}}^{2}\n", "$$\n", "*Среднеквадратическое отклонение*, *стандартное отклонение* или *стандартный разброс* - квадратный корень из дисперсии, равный $\\sigma$\n", "\n", "Стандартное отклонение измеряется в тех же единицах, что и сама случайная величина, а дисперсия измеряется в квадратах этой единицы измерения." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data[['Pclass', 'Age']].groupby('Pclass').std()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Квантили**\n", "\n", "Квантиль - значение, которое заданная случайная величина не превышает с фиксированной вероятностью. Если вероятность задана в процентах, то квантиль называется процентилем или перцентилем.\n", "\n", "Например, фраза «для развитых стран 99-процентиль продолжительности жизни составляет 100 лет» означает, что ожидается, что 99 % людей проживут не более, чем 100 лет.\n", "\n", "Относительно нашего датасета фраза \"75%-перцентиль возраста пассажиров Титаника равна 38 лет\" означает, что 75% пассажиров были не старше 38 лет.\n", "\n", "Вопросы знатокам:\n", "- Как еще можно назвать 50%-перцентиль? 🤔\n", "- Что значит фраза \"75%-перцентиль стоимости билетов на Титаник была равна 31\"? 🤔" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data.Age.describe()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data.Age.hist(bins=25)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Описание массива нечисловых данных**\n", "\n", "В качестве базовой описательной статистики для категориальных признаков можно использовать следующий набор характеристик:\n", "\n", "- количество уникальных представителей массива\n", "\n", "- частоты встречаемости этих представителей\n", "\n", "- наиболее часто встречающиеся представители (мода распределения)\n", "\n", "- наиболее редко встречающиеся представители" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data.Pclass.unique()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data.Pclass.value_counts()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data.Pclass.mode()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## И еще чуть-чуть! Работа с пропущенными значениями." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Вернемся к info(). Мы видим, что из 891 наблюдения у нас только 714 ненулевых значений. Значит, у этих пассажиров возраст неизвестен. Ваши идеи, что можно с этим сделать?" ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Int64Index: 183 entries, 1 to 889\n", "Data columns (total 15 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 PassengerId 183 non-null int64 \n", " 1 Survived 183 non-null int64 \n", " 2 Pclass 183 non-null int64 \n", " 3 Name 183 non-null object \n", " 4 Sex 183 non-null object \n", " 5 Age 183 non-null float64\n", " 6 SibSp 183 non-null int64 \n", " 7 Parch 183 non-null int64 \n", " 8 Ticket 183 non-null object \n", " 9 Fare 183 non-null float64\n", " 10 Cabin 183 non-null object \n", " 11 Embarked 183 non-null object \n", " 12 FamilySize 183 non-null int64 \n", " 13 Alone 183 non-null int64 \n", " 14 Title 183 non-null object \n", "dtypes: float64(2), int64(7), object(6)\n", "memory usage: 22.9+ KB\n" ] } ], "source": [ "data.info()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data['Age'].median() # вспомним какая у нас медиана" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data['Age_Median'] = data['Age'].fillna(data['Age'].median()) # сохраняю результат заполнения в новую колонку" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data.head() # А у всех остальных - их нормальный возраст." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Или можно просто удалить пропущенные значения:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data.dropna()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 2 }