Building Embeddings

Embeddings are extracted from the images by training a convolutional autoencoder.

The models available are listed in models. The default autoencoder is a 3-layer convolutional autoencoder, `~simages.BasicAutoencoder:

from simages import EmbeddingExtractor

extractor = EmbeddingExtractor(image_dir)

extractor allows identifying images corresponding to the embeddings.

For example, if extractor.duplicates() returns pairs [2, 3], the images corresponding to embeddings 2 and 3 can be viewed with simages.extractor.EmbeddingExtractor.image_paths():

EmbeddingExtractor.image_paths(indices, short=True)[source]

Get path to image at index of eval/embedding

Parameters:
  • Union[int,list] (indices) – indices of embeddings in dataset
  • short (bool) – truncate filepath to 30 charachters
Returns:

paths to images in image folder
Return type:

paths (str or list of str)

After building the embeddings, the embeddings can be extracted with:

emeddings = extractor.embeddings

Extraction is performed by EmbeddingExtractor:

class simages.extractor.EmbeddingExtractor(input: Union[str, numpy.ndarray], num_channels: int = 3, num_epochs: int = 2, batch_size: int = 32, show: bool = False, show_path: bool = False, show_train: bool = False, z_dim: int = 8, metric: str = 'cosine', model: Optional[torch.nn.modules.module.Module] = None, db: Optional = None, **kwargs)[source]

Bases: object

Extract embeddings from data with models and allow visualization.

trainloader
Type:torch loader
evalloader
Type:torch loader
model
Type:torch.nn.Module
embeddings
Type:np.ndarray
__init__(input: Union[str, numpy.ndarray], num_channels: int = 3, num_epochs: int = 2, batch_size: int = 32, show: bool = False, show_path: bool = False, show_train: bool = False, z_dim: int = 8, metric: str = 'cosine', model: Optional[torch.nn.modules.module.Module] = None, db: Optional = None, **kwargs)[source]

Inits EmbeddingExtractor with input, either str or np.ndarray, performs training and validation.

Parameters:
  • input (np.ndarray or str) – data
  • num_channels (int) – grayscale = 1, color = 3
  • num_epochs (int) – more is better (generally)
  • batch_size (int) – number of images per batch
  • show (bool) – show closest pairs
  • show_path (bool) – show path of duplicates
  • show_train (bool) – show intermediate training results
  • z_dim (int) – compression size
  • metric (str) – distance metric for scipy.spatial.distance.cdist() (eg, euclidean, cosine, hamming, etc.)
  • model (torch.nn.Module, optional) – class implementing same methods as BasicAutoencoder
  • db_conn_string (str) – Mongodb connection string
  • kwargs (dict) –
get_image(index: int) → torch.Tensor[source]
train()[source]

Train autoencoder to build embeddings of dataset. Final embeddings are created in eval().

eval()[source]

Evaluate reconstruction of embeddings built in train.

duplicates(n: int = 10, quantile: float = None) → Tuple[numpy.ndarray, numpy.ndarray][source]

Identify n closest pairs of images, or quantile (for example, closest 0.05).

Parameters:
  • n (int) – number of pairs
  • quantile (float) – quantile of total combination (suggested range: 0.001 - 0.01)
static channels_last(img: numpy.ndarray) → numpy.ndarray[source]

Move channels from first to last by swapping axes.

show(img: Union[torch.Tensor, numpy.ndarray], title: str = '', block: bool = True, y_labels=None, unnormalize=True)[source]

Plot img with title.

Parameters:
  • img (torch.Tensor or np.ndarray) – Image to plot
  • title (str) – plot title
  • block (bool) – block matplotlib plot until window closed
show_images(indices: Union[list, int], title='')[source]

Plot images (from validation data) at indices with title

image_paths(indices, short=True)[source]

Get path to image at index of eval/embedding

Parameters:
  • Union[int,list] (indices) – indices of embeddings in dataset
  • short (bool) – truncate filepath to 30 charachters
Returns:

paths to images in image folder
Return type:

paths (str or list of str)
show_duplicates(n=5, path=False) -> (<class 'numpy.ndarray'>, <class 'numpy.ndarray'>)[source]

Show duplicates from comparison of embeddings. Uses closely package to get pairs.

Parameters:
  • n (int) – how many closest pairs to identify
  • path (bool) – Plot pairs of images with abbreviated paths
Returns:

pairs as indices distances (np.ndarray): distances of pairs
Return type:

pairs (np.ndarray)
unnormalize(image: torch.Tensor) → torch.Tensor[source]

Unnormalize an image.

Parameters:image (torch.Tensor) –
Returns:image (torch.Tensor)
decode(embedding: Optional[numpy.ndarray] = None, index: Optional[int] = None, show: bool = False, astensor: bool = False) → numpy.ndarray[source]

Decode embeddings at index or pass embedding directly

Parameters:
  • embedding (np.ndarray, optional) – embedding of image
  • index (int) – index (of validation set / embeddings) to decode
  • show (bool) – plot the results
  • astensor (bool) – keep as torch.Tensor
Returns:

reconstructed image from embedding
Return type:

image (np.ndarray or torch.Tensor)